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Executive Summary 

When an effective principal is at the helm of a school, students benefit (Branch, Hanushek, & 
Rivkin, 2012). Principals play a critical role in establishing a school's climate and culture and in 
selecting and developing teachers, among other roles. Indeed, school leadership is second only 
to direct classroom teaching among school-level influences on achievement (Hallinger & Heck, 
1998; Leithwood, Louis, Anderson, & Wahlstrom, 2004). Although there may be little 
disagreement that good principals make a difference, what is less clear is how to systematically 
prepare good principals. 

The George W. Bush Institute has made school principals a focus of its education reform initiative 
through the Alliance to Reform Education Leadership (AREL) program. The AREL program was 
launched by convening a network of 28 innovative principal preparation programs for 3 years to 
learn from the programs, help the programs learn from each other, and spotlight the work done 
by these programs. With this current study, the Bush Institute sought to go beyond sharing 
information about best practices in principal preparation and connect information about program 
graduates to student outcomes. Specifically, this study evaluated the impact of five AREL 
Network programs on student achievement. We also gathered information on graduates' 
perceptions of their programs. The five principal preparation programs were selected for 
inclusion in the study based on a set of criteria developed to reflect the best available theory and 
research on promising practices in principal preparation. 

The study was designed to address the following research questions (RQs): 

1. What are the characteristics of the selected preparation programs and the district- 
provided supports to program graduates? 

2. What are the characteristics of schools in which program graduates were placed? 

3. What effect do program graduates in their first placements as principals have on student 
achievement and other outcomes? 1 - 2 

3a. How much variation is there in program graduates' effects on student achievement? 

4. What effect do program graduates with varying levels of experience as principals have on 
student achievement? 

Assessing the impact of principal preparation programs on student outcomes is not without its 
challenges. Some of the challenges are as follows: principals affect achievement indirectly, and it 
may take time for leadership to have a measurable effect on student learning; the single-year 


1 In one district (E), complete data to address RQ3 and RQ4 were unavailable. In this case, RQ3 was 
modified as described in the following sections, and RQ4 was omitted. 

2 The original analysis plans included analysis of achievement, school climate, and graduation rate 
outcome data— where possible. Ultimately, however, usable school climate data were not consistently 
available in the districts we studied. In addition, most programs had no principals or very few principals 
assigned to high schools, and initial feasibility analysis of graduation rates suggested that we would not be 
able to detect effects on 4-year cohort graduation rates, so the analysis of graduation rates was dropped. 
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impact of principals' effects on student achievement may be relatively small (Clifford, Behrstock- 
Sherratt, & Fetters, 2012); the number of graduates from any single program that have been placed 
and retained as principals may be small; and there may be factors in the district or elsewhere that 
cannot be disentangled from the effects of preparation programs, such as additional supports that 
districts provide to new principals or even the culture of a district. These challenges and others are 
described in more detail in the report. 

These challenges do not lessen the importance of tracking program completers after they graduate 
and gathering and rigorously analyzing data about their placement, retention, and their school and 
student outcomes, including achievement. They do, however, point to the need to consider multiple 
factors in determining the overall effectiveness of any single preparation program. 

Key Findings 

Districts and preparation programs lacked high-quality data on principal characteristics and 
placements. High-quality data about principal preparation, experience, and assignment were not 
readily available within the districts where program graduates were placed. For example, one 
district was unable to provide information on principal experience (in or outside the district). 
Other districts did not have data about principal preparation, so we relied on data from the 
preparation programs themselves about their graduates. Data that describe principals' 
participation in district professional learning, support, or other leadership initiatives (e.g., grant 
programs targeting leadership support, mentoring programs, and coaching) were typically not 
centrally collected or complete. In addition, consistent or accurate data about student outcomes 
other than achievement were not readily available, including data on school climate. 

Selected program graduates had generally positive perceptions of program coursework and 
hands-on experiences, but they have mixed perceptions of district supports and ongoing 
supports from their programs. The results of 68 interviews conducted with principals who 
graduated from the five selected programs included generally positive feedback on key aspects 
of the programs, such as coursework, hands-on experiences, and the rigor of selection into the 
programs. For example, approximately three fourths of the respondents found their coursework 
to be supportive of their practice in the principal role to "a great extent." However, graduates 
described variety in the supports received once they were placed and in their perceptions of the 
supports. They had mixed views on the usefulness of district supports, with some respondents 
noting very minimal or ineffective district support, and others praising extensive support from 
the district. The majority of respondents across three programs found the support they received 
from programs (e.g., coaching) to be "greatly beneficial" to their practice in the principal role, 
whereas some other respondents described the coaching support they received as only 
"somewhat beneficial." 

We found little consistent evidence that student achievement in schools led by program 
graduates is better (or worse) than student achievement in similar schools led by graduates of 
other programs. Using two different analytic approaches (one that focused on principals newly 
placed during the time period of the study and one that included all principals from the selected 
programs serving as principals in schools during the same period), we found little consistent 

2 


evidence that graduates from any program were more or less effective at fostering student 
achievement than principals from other programs. It is important to note that this finding does not 
indicate that these programs are not effective. It does mean that within the time period that we 
studied, we were unable to identify average effects of the programs on student achievement. 

Significant variation occurred in effectiveness among principals from selected and other 
programs. Although we observed little consistent evidence about average program effects, we 
observed significant variation in effectiveness among inexperienced principals— from both 
selected and other programs. In some schools led by one of the selected program graduates, 
academic performance was well above the average for new principals, but in other schools, it was 
well below the average. 

Taken together, these findings suggest that focusing on how to reduce variation in the 
performance_of graduates through training, selection, or other means or how to systematize or 
better tailor supports may be the keys to success in preparing effective school leaders. 
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Introduction 


When an effective principal is at the helm of a school, students benefit— by one estimate, students 
can gain as much as 7 months of additional learning in one school year (Branch, Hanushek, & 
Rivkin, 2012). Principals play a critical role in establishing a school's climate and culture and in 
selecting and developing teachers, among other roles. Indeed, school leadership is second only 
to direct classroom teaching among school-level influences on achievement (Hallinger & Heck, 
1998; Leithwood, Louis, Anderson, & Wahlstrom, 2004). 

Although there may be little disagreement that good principals make a difference, what is less 
clear is how to systematically prepare good principals. States typically require principals to 
complete a preparation program to obtain an administrative certification, although the criteria 
for the design of these preparation programs vary. Historically, universities have provided 
principal preparation programs, as is true with teacher preparation, and 48 states require 
principal candidates to obtain a graduate degree in education or another field to be certified (26 
states require a graduate degree in education). Across the United States, as many as 700 principal 
preparation programs are preparing and certifying principals to lead our nation's schools 
(Anderson & Reynolds, 2015; Wallace Foundation, 2016). All states allow nonprofit or alternative 
programs to prepare principals, and 35 state laws or rules specify that nontraditional programs 
are required to meet the same program quality criteria that other, traditional programs must meet 
(Wallace Foundation, 2016). 

Current approaches employed by university-based principal preparation programs, which 
continue to train the majority of future principals, often are not viewed as being sufficient to 
prepare principals to lead schools. For example, Darling-Hammond, LaPointe, Meyer son, Orr, 
and Cohen (2007) noted that educators in a variety of roles, including principals themselves, 
reported that universities generally offer principal preparation programs that are "out of touch 
with the real-world complexities and demands of school leadership" (p. 5), have a low bar for 
admissions, have programming that is based on outdated research, have curricula that are not 
aligned to current research-based standards, and lack meaningful experiential learning. In 
another study, 69% of the principals and 80% of the superintendents reported that typical 
leadership programs are not aligned to the realities of current districts (Farkas, Johnson, Duffett, 
& Foleno, 2001). 

Study Background 

The George W. Bush Institute has made school principals a focus of its education reform initiative 
through the Alliance to Reform Education Leadership (AREL) program. Rooted in President and 
Mrs. Bush's belief that "excellent schools must first have excellent leaders," the AREL program 
was developed to improve the way U.S. principals are prepared and supported. The AREL 
program was launched by convening a network of 28 innovative principal preparation programs 
for 3 years to learn from the programs, help the programs learn from each other, and spotlight 
the work done by these programs. 
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Convening the AREL Network was a first step into understanding expert opinion on best 
practices in principal preparation. With the current study, the Bush Institute sought to go beyond 
sharing information about best practices in principal preparation and connect information about 
program graduates to student outcomes. Specifically, this study evaluated the impact of five 
AREL Network programs on student achievement. We also gathered information on graduates' 
perceptions of their programs. 

Most discussion and research on the use of student test scores to assess educator preparation has 
focused on teacher preparation programs. With a few notable exceptions (e.g.. Gates et al. ; 2014), 
few studies have analyzed the impact of principal preparation programs on student learning. 
This study aimed to extend our knowledge about program impact by evaluating multiple 
programs and implementing a rigorous methodology that could potentially serve as a model for 
future studies or improvement efforts. 

Assessing the impact of principal preparation programs on student outcomes is not without its 
challenges. Some of the challenges are as follows: principals affect achievement indirectly, and it 
may take time for leadership to have a measurable effect on student learning; the single-year 
impact of principals' effects on student achievement may be relatively small (Clifford et al., 2012); 
the number of graduates from any single program that have been placed and retained as 
principals may be small; and there may be factors in the district or elsewhere that are difficult to 
disentangle from the effects of preparation programs, such as additional supports that districts 
provide to new principals or even the culture of a district. These challenges and others are 
described in more detail in the Methods section. 

These challenges do not lessen the importance of tracking program completers after they 
graduate and gathering and rigorously analyzing data about their placement, retention, and their 
school and student outcomes, including achievement. They do, however, point to the need to 
consider multiple factors in determining the overall effectiveness of any single preparation 
program. 

Study Overview 

Five AREL Network principal preparation programs were selected for inclusion in the study 
based on a set of criteria developed to reflect the best available theory and research on promising 
practices in principal preparation. The study was designed to address the following RQs: 

1. What are the characteristics of the selected preparation programs and the district- 
provided supports to program graduates? 

2. What are the characteristics of schools in which program graduates were placed? 
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3. What effect do program graduates in their first placements as principals have on student 
achievement and other outcomes? 3 - 4 

3a. How much variation is there in program graduates' effects on student achievement? 

4. What effect do program graduates with varying levels of experience as principals have on 
student achievement? 

We used multiple approaches to address these questions (Table 1). The Methods section provides 
more detail on the methods and data. 


Table 1. Research Questions and Data Collection Methods 


Research Question 

Data Collection and Analysis 

RQ1. What are the characteristics 
of the selected preparation 
programs and the district- 
provided supports to program 
graduates? 

■ Interviews with key preparation program administrators, a 
sample of graduates from each program, and district staff 
responsible for principal recruitment and development in 
districts where graduates were placed. 

■ Review of program documentation. 

RQ2. What are the characteristics 
of schools in which program 
graduates were placed? 

Descriptive analysis of publicly available aggregate school-level 
demographic data for schools in the districts matched to data 
from selected programs about where their graduates served as 
principals. These data were then matched to placement data 
provided by the programs to identify where their graduates 
served as principals during the period of the study. Finally, we 
analyzed student test scores in reading/English language arts 
(ELA) and mathematics in Grades 3-8 and matched to data 
about where the programs placed their graduates. 

RQ3. What effect do program 
graduates in their first placements 
as principals have on student 
achievement and other outcomes? 

3a. How much variation is there in 
program graduates' effects on 
student achievement? 

Quantitative analysis of student test scores in reading/ELA and 
mathematics in Grades 3-8 and matched to data about where 
selected programs placed their graduates (focus on first-time 
principals only). 

RQ4. What effect do program 
graduates with varying levels of 
experience as principals have on 
student achievement? 

Quantitative analysis of student test scores in reading/ELA and 
mathematics in Grades 3-8 and matched to data about where 
selected programs placed their graduates (includes all principals 
from the program serving in schools during the study period). 


3 In one district (E), complete data to address RQ3 and RQ4 were unavailable. In this case, RQ3 was 
modified as described in the following sections, and RQ4 was omitted. 

4 The original analysis plans included analysis of achievement, school climate, and graduation rate 
outcome data— where possible. Ultimately, however, usable school climate data were not consistently 
available in the districts we studied. In addition, most programs had no principals or very few principals 
assigned to high schools, and initial feasibility analysis of graduation rates suggested that we would not be 
able to detect effects on 4-year cohort graduation rates, so the analysis of graduation rates was dropped. 
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Organization of the Report 

This remainder of this report is divided into the following major sections: 

• Key findings 

• Methods 

• Findings 

• Discussion and implications 

Following an overview of the key findings, we describe all the methods, beginning with our 
approach to selecting and describing the programs and collecting interview data from program 
administrators and principals. We also give an overview of our quantitative analysis methods, 
with more detailed descriptions available in Appendix F. We close the quantitative methods 
section with an overview of the methodological challenges of this work. 

Next, we give an overview of the findings about program characteristics and district supports 
(RQ1), including program descriptions and principal perception data. This section also provides 
descriptive statistics on each district where graduates were placed (RQ2) and follows with the 
findings on graduate performance for all five programs (RQ3 and RQ4). 

The report closes with a discussion of the implications of our findings and suggestions for future 
research. The appendices include program selection criteria, the program documentation request, 
principal and program administrator interview protocols, additional principal perception 
findings, more detailed quantitative methods, and more detailed empirical findings. 
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Key Findings 

Districts and preparation programs lacked high-quality data on principal characteristics and 
placements. High-quality data about principal preparation, experience, and assignment were not 
readily available within the districts. For example, one district was unable to provide information 
on principal experience (in or outside the district). Other districts did not have data about principal 
preparation, so we relied on data from the preparation programs themselves about their graduates. 
Data that describe principals' participation in district professional learning, support, or other 
leadership initiatives (e.g., grant programs targeting leadership support, mentoring programs, and 
coaching) were typically not centrally collected or complete. In addition, consistent or accurate data 
about student outcomes other than achievement were not readily available. 

Selected program graduates had generally positive perceptions of program coursework and 
hands-on experiences, but they have mixed perceptions of district supports and ongoing 
supports from their programs. The results of 68 interviews conducted with principals who 
graduated from the five selected programs included generally positive feedback on key aspects 
of the programs, such as coursework, hands-on experiences, and rigor of selection into the 
programs. For example, approximately three fourths of the respondents found their coursework 
to be supportive of their practice in the principal role to "a great extent." However, graduates 
described variety in the supports received once they were placed and in their perceptions of the 
supports. They had mixed views on the usefulness of district supports, with some respondents 
noting very minimal or ineffective district support, and others praising extensive support from 
the district. The majority of respondents across three programs found the support they received 
from programs (e.g., coaching) to be "greatly beneficial" to their practice in the principal role, 
whereas some other respondents described the coaching support they received as only 
"somewhat beneficial." 

We found little consistent evidence that student achievement in schools led by program 
graduates is better (or worse) than student achievement in similar schools led by graduates of 
other programs. Using two different analytic approaches (one that focused on principals newly 
placed during the time period of the study and one that included all principals from the selected 
programs serving as principals in schools during the same period), we found little consistent 
evidence that graduates from any program were more or less effective at fostering student 
achievement than principals from other programs. 5 It is important to note that this finding does not 
indicate that these programs are not effective. It does mean that within the time period that we 
studied, we were unable to identify average effects of the programs on student achievement. 

Significant variation occurred in effectiveness among principals from selected and other 
programs. Although we observed little consistent evidence about average program effects, we 
observed significant variation in effectiveness among inexperienced principals— from both 


5 Because of data limitations in one district, we focus here on findings from four of the five originally 
selected programs. The results for that district (District E) can be found in the section on quantitative 
results for District E. 
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selected and other programs. In some schools led by one of the selected program graduates, 
academic performance was well above the average for new principals, but in other schools, it was 
well below the average. 

Taken together, these findings suggest that focusing on how to reduce variation in the 
performance of graduates through training, selection, or other means, or how to systematize or 
better tailor supports may be the keys to success in preparing effective school leaders. 
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Methods 


Program Selection Process 


To develop an initial pool of potential programs to be evaluated, we first looked at those 
participating in the AREL Network and then considered recommendations from experts in the 
field and professional associations, such as the University Council for Educational 
Administration (UCEA). We developed a set of criteria reflecting the best available theory and 
research on promising practices in principal preparation and then rated each program (see 
Appendix A). A review of the research on what experts consider promising principal preparation 
programs generally characterizes the features of these programs into several categories: 
alignment to research-based competencies, rigorous selection criteria, the inclusion of 
experiential learning, on-the-job support (mentoring, coaching, and professional development), 
and strong district-program partnerships (Darling-Hammond et al., 2007; George W. Bush 
Institute, 2014; Shelton, 2012). The selection criteria were as follows: 

• Program Alignment to Research-Based Competencies. The program's structure and 
curriculum is organized according to a set of research-based standards and practices and 
displays alignment to these research-based competencies. 

• Experiential Learning. The program has student learning experiences (including coursework 
and other learning experiences) that expose students to the problems of school leadership 
practice, with the intent to build practical and technical knowledge. Learning experiences are 
scaffolded, moving from classroom or online learning simulations to internship experiences 
where students lead all or a significant portion of a school's operations. 

• High-Quality, Rigorous Recruitment and Selection. The program engages in vigorous 
recruitment of high-ability candidates with experience as expert, dynamic teachers and a 
commitment to instructional improvement. 

• Early Years on-the-job Support. The program includes intensive on-the-job support (i.e., 
induction) for program graduates who are principals, such as mentoring, coaching, or a 
peer network, for at least 1 year after graduation. 

• Partnerships for Excellence. The program's administrative structure engages district 
personnel in coteaching classes, serving as sites for learning, providing feedback on 
graduate quality, or curriculum review and alignment. 

• Evidence of Effects. The program seeks out evidence of effects on advancing leadership 
practice, school culture, or student learning that have been developed through rigorous 
study. The program also has established routines for collecting program effectiveness 
information and uses data to adjust programs. 

In addition, we selected programs that placed most or all of their graduates in one district to 
facilitate data analysis. Ultimately, based on these criteria, of the 11 programs reviewed, five 
programs were selected for inclusion in the study. Complete documentation of our selection 
criteria is in Appendix A. 
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Program Document Review and Administrator Interviews 


To learn more about the characteristics of the five selected programs, including how they align to 
the selection criteria, we requested documentation from the selected programs (Table 2) and 
conducted interviews with one or more program administrators from each program. These 
interviews produced a detailed description of each program, which was reviewed for accuracy by 
each program. The program documentation request and interview protocol are in Appendix B and 
Appendix C. A synthesis of the findings from the document review and interviews, including 
detailed examples from particular programs as they relate to each selection criterion, can be found 
in the results section. 

Table 2. Types of Documentation Requested of the Programs 

Program descriptions 

Admission and graduation statistics 

Placement data and statistics 

Recruitment materials 

Coursework descriptions and syllabi 

Program standards and competencies 

Documentation of program alignment to standards 

Program planning documentation 

Faculty information 

Materials describing clinical residencies or internships 
Descriptions of coaching or mentoring programs 

Information documenting partnerships between programs and districts 
Evaluation data or reports 


Principal Interviews 

To further explore RQ1 regarding program characteristics and better understand the types of 
supports provided to graduates of the selected programs, both by the program and the partner 
district, we interviewed a sample of principals who graduated from each selected program. The 
purpose of these principal interviews was fourfold: 

• Learn more about the principals' background and experience in education. 

• Determine if graduates' schools or districts were participating in any special initiatives. 

• Understand the supports that principals received from the program, the district, and any 
other sources. 

• Learn more about how the principal graduates have applied material learned in the 
program in their roles as school leaders. 

These interviews were not intended to compare the experiences of principals trained by the 
selected programs with the experiences of principals trained by other programs; they were 
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intended to provide only more detailed descriptive information on the practices of the selected 
programs. The complete interview protocol is in Appendix D. 

Those interviewed were selected from a complete list of all graduates of each program obtained 
from the program itself and, when available, cross-checked by data provided by the partner 
district. This list included both principals who graduated from the program and currently serve 
as principals and graduates that served as principals in the district but then left their initial 
principal position. 

Telephone interviews were conducted from October 2014 through May 2015 and were 
approximately 60-90 minutes in length. Researchers recorded and transcribed the interviews and 
then coded and analyzed the interview transcripts and notes using NVivo software. 6 


6 Interviews were recorded only when the respondent provided consent. In the event that a respondent 
did not want to be recorded, the analysis was based on the interviewer's notes. 
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Student Achievement Analysis 


To address RQ3 and RQ4 regarding the effects of program graduates on student achievement and 
school climate (where data were available), we employed several different analytic strategies 
based on the available data and the specifics of the research questions. We describe each analytic 
approach in the following sections. 

In each case, the program effect for RQ3 and RQ4 is a school-level factor. We assume that 
principals who are trained by strong preparation programs will emerge with good leadership 
practices (which will, in turn, be affected by the supports they receive from their programs or 
districts, as well as by their own leadership experience (Figure 1). We expect these practices to 
lead to school-level changes that ultimately result in school-level improvements in student 
achievement. 

Figure 1. Logic Model for Principal Preparation Program Effects on Student Outcomes 


Strong 

Principal 
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Program 

(effective 
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Each of the five principal preparation programs mentioned earlier is unique, as are the districts 
with which the principal preparation programs partner. Therefore, our research design specifies 
a separate analysis for each program. 

Based on within-school estimates of principal value-added effects, Grissom, Kalogrides, and Loeb 
(2015) estimate that a one standard deviation (SD) increase in principal effectiveness is associated 
with an increase in student achievement equivalent to between 0.058 and 0.084 SD of the test 
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score distribution in mathematics and between 0.038 and 0.065 SD of the test score distribution 
in reading. These measures provide a benchmark by which to assess the relative effectiveness of 
principals trained by one of the programs of interest. 

Estimating Effects of Newly Placed (Inexperienced) Principals From 

Selected Programs 

Defining the Treatment 

For RQ3, the treatment is being assigned a principal newly trained by one of the principal 
preparation programs of interest. The counterf actual— what would have happened in the absence 
of treatment— is what would have happened at that same school had a principal from one of the 
preparation programs of interest not been assigned to the school. 

For RQ3a on variation in program graduates' effects on student achievement, the treatment is 
being assigned a principal newly trained by one of the principal preparation programs of interest, 
and the counterfactual is what would have happened at that same school if a principal from a 
different program had been assigned at the same time. 

Our research design measures the difference between what actually happened at schools during 
years they were led by a principal from one of the programs of interest and what would have 
happened in the counterfactual. This difference in student outcomes at the school is interpreted 
as a causal effect of principals from the programs of interest on student outcomes. The theoretical 
framework for this design (the potential outcomes framework) derives from Rubin's (1974) causal 
model. 

The following further clarify the treatment: 

• Our analysis cannot disentangle the process by which individuals are selected (or self- 
select) into preparation programs. For example, some programs may be better at selecting 
individuals who become more effective principals (e.g., those with greater content 
knowledge or other relevant skills) than others. A program may then appear to be more 
effective because it selects highly qualified applicants, even though the program does little 
to improve the skills of those it prepares. However, the process by which the principal 
preparation program selects among its applicants may be viewed as an important 
component of the program. The goal of this project is to determine the effect of principals 
from the programs of interest. The project will not explain why those principals are more 
effective. 

• In some schools in our study, a principal trained by one of the programs of interest is 
placed at a school, serves for a number of years, leaves the school, and is replaced by 
another principal from the same program. This creates a problem for our identification 
strategy: Should the first year the second principal leads the school be considered the first 
year the school is treated by an inexperienced principal from one of the programs of 
interest or a continuation of the leadership from the previous principal trained by one of 
the programs of interest? Because neither approach would provide a clean measure of the 
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number of years the school had been led by an inexperienced principal from one of the 
programs of interest, we drop from our sample all observations where a school was led 
by a second principal from one of the programs of interest. 

• The goal of this analysis is to evaluate the effectiveness of principals trained by one of the 
five selected programs. However, if many comparison-group principals also are trained 
by other high-quality programs that serve the district, we might not expect to find 
significant differences in effectiveness between principals from programs of interest and 
principals from other programs. 

• In some cases, several years may pass between the time an individual completes one of 
the programs of interest and the time the individual first becomes a principal. We did not 
control for the number of years that pass between the time an individual completes a 
principal preparation program and the time the individual becomes a principal. 
Individuals who do not secure a principal position until several years after completing 
their principal preparation program might be different, on average, than individuals who 
secure a principal position shortly after completing a principal preparation program. For 
example, individuals who do not secure a position until several years after completing the 
program might have been less effective than principals who secured a position 
immediately for reasons that have nothing to do with the span of time between 
completing the program and securing a position. Therefore, controlling for the number of 
years that pass between the time an individual completes a preparation program and the 
time the individual secures a principal position could remove important variation in 
principal quality from our estimates of the relative effectiveness of the programs of 
interest. To the extent that the potential effect of the program of interest decays between 
the time the individual completes the program and the time the individual receives his or 
her first principal assignment, however, our estimates of the effect of the programs of 
interest on achievement at schools will be attenuated. 

Analytic Approach 

RQ3 asks about the effect of program graduates in their first placements as principals after being 
trained by the selected program. To evaluate the effect of being assigned a principal newly trained 
by one of the selected programs on outcomes at that school (compared with schools that were 
assigned an inexperienced principal not trained by one of the selected programs), we identified 
all schools in the participating districts that were assigned a principal who had no experience as 
a principal in that district prior to placement. Most districts were unable to provide information 
on principals' experience outside the district, so our analysis was able to identify only those 
principals with no experience within the selected program partner district. The implication of this 
fact is that although we refer to this analysis as addressing inexperienced principals, it is possible 
that some comparison group principals did have leadership experience outside the district, and 
we simply observed their first placements within the district. 7 


7 It also is possible that both treatment and comparison group principals held school leadership positions, 
such as assistant principal or teacher leader, prior to becoming a principal. 
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The analysis focused only on those schools that received an inexperienced principal during the 
2011-12, 2012-13, 2013-14, or 2014-15 school years. It includes both treatment schools that 
received an inexperienced principal from one of the selected programs and comparison schools 
that received an inexperienced principal who did not attend one of the selected programs. We 
included only those schools for which outcome data were available every year from 2008-09 
through 2014-15. 

The gold standard for determining program impact in education research is the randomi z ed 
experiment. Using this research design, principals would be randomly assigned to schools, and 
outcomes of students in schools led by principals from one of the selected programs would be 
compared with the outcomes of students in other schools. Because such a design is impractical in 
this case, we employed a quasi-experimental analytic strategy— a within-school comparative 
interrupted time series (CITS). CITS designs have been shown to produce results similar to those 
of other quasi-experimental designs (Somers, Zhu, Jacob, & Bloom, 2013) and randomized 
controlled trials (St. Clair, Cook, & Hallberg, 2014, 2016). 

In our CITS design, student outcomes at each school were expected to continue along the same 
linear trend followed prior to the arrival of an inexperienced principal, along with any additional 
changes in student outcomes one might expect because of changes in the characteristics of 
students attending the school. The expectation that student outcomes will continue along their 
previous linear trend is the "time series" component of CITS. 

The CITS analysis included schools that received an inexperienced principal from one of the 
selected programs as well as other schools that received inexperienced principals who were 
graduates of other programs (i.e., not the selected programs). Following the arrival of the new 
principals, changes in outcomes at schools that received an inexperienced principal from one of 
the selected programs were compared with changes in outcomes at schools that received other 
inexperienced principals. The estimated average change in outcomes following the arrival of 
inexperienced principals in schools is the "interrupted" component of CITS. Finally, the 
difference between the average change in outcomes between schools that received an 
inexperienced principal from a selected program and the average change in outcomes among 
schools that received other inexperienced principals is the "comparative" component of CITS. 

The comparison schools— schools that received a new principal who was not trained by one of 
the selected programs— were included in the analysis to provide an estimate of a counterfactual 
for the treatment schools during the "post" period (i.e., following the placement of the 
inexperienced principal). For example, if the achievement in comparison schools dipped 
following the arrival of the inexperienced principal, then one might expect achievement in 
treatment schools to dip following the arrival of the inexperienced principal as well. Similarly, if 
achievement in comparison schools increased following the arrival of the inexperienced 
principal, then one might expect achievement in treatment schools to increase as well. 
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Two assumptions that must be true for the CITS design to produce unbiased estimates of the 
relative effectiveness of inexperienced principals from selected programs are as follows: 

1. In a typical school included in the analysis, if an inexperienced principal had not been 
placed in the school, the trend in outcomes at the school would have continued along its 
previous linear path. 

2. If an inexperienced principal from one of the selected programs had not been placed in 
the school, but an inexperienced principal from another program had been placed there 
instead, the change in outcomes one could expect to observe in that school would be the 
same as the typical change in outcomes in following the arrival of an inexperienced 
principal in comparison schools. 

The analysis covers the years 2008-09 through 2014-15. To include at least three pretreatment years 
of data in the analysis, we examined outcomes for four cohorts of schools (Figure 2). The analysis 
for Cohort 1 included three pretreatment years and four treatment years; Cohort 2 included four 
pretreatment years and three treatment years; Cohort 3 included five pretreatment years and two 
treatment years; and Cohort 4 included six pretreatment years and one treatment year. 


Figure 2. Overview of Graduate Cohorts Included in the Analysis 
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Principal Attrition in Our Analysis of Inexperienced Principals 

Our analysis of inexperienced principals includes only those principals who were placed during 
2011-12, 2012-13, or 2013-14 and who stayed at the same school through 2014-15. Not all 
inexperienced program principals who were placed in 2011-12, 2012-13, or 2013-14 stayed at the 
same school through 2014—15. For example, a principal from one of the selected programs might 
have been placed in a school in 2011-12 and then left that school after 2012-13. If the average 
effectiveness of program principals who leave their schools before 2014—15 is different from the 
average effectiveness of program principals who stay through 2014—2015, our findings will not be 
generalizable to the group of principals (program and nonprogram) who change schools. Similarly, 
if the average effectiveness of nonprogram principals who leave their schools before 2014-15 is 
different from the average effectiveness of nonprogram principals who stay through 2014—2015, our 
findings will not be generalizable to the group of inexperienced principals (program and 
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nonprogram) who change schools. We considered several strategies to handle these situations, but 
ultimately decided to drop these schools from the analysis. 

One possible solution to this problem would have been to match every inexperienced program 
principal who left before 2014-15 with a comparison-group principal who started and left at the 
same times as the program principal and whose school had similar characteristics as the program 
principal's school. This would allow us to compare two principals, one program and one 
nonprogram, who led similar schools, became principals at the same time, and remained at their 
schools for the same amount of time. Unfortunately, we could not always find comparison-group 
principals who met all these criteria. 

Another strategy would be to simply retain the school in the analysis through 2014-15 and continue 
to look at outcomes at the school after the principal had left. With this approach, however, our 
models would not clearly distinguish between the effectiveness of the principal who left and the 
effectiveness of that principal's replacement. If a principal's departure and the arrival of a new 
principal is disruptive to the school, this strategy might affect the achievement outcome we plan to 
study. 

In the end, we decided that the best strategy would be to simply remove these principals and 
schools from the analysis. This strategy limits the generalizability of our findings to inexperienced 
principals who stayed in their schools through the available time period (i.e., for either 1, 2, 3, or 
4 years depending on when they were first placed). Note that this potential shortcoming applies 
only to our analysis of inexperienced principals; it does not apply to our analysis of all principals, 
which is described in greater detail as follows. 

Table 3 shows the numbers of selected program principals included in the analysis of inexperienced 
principals and the numbers who were dropped because they did not remain at their schools. 
Overall, only 5 of the 60 selected program principals were excluded from the analysis because they 
did not remain at their schools. Overall, sample sizes among selected programs are likely more 
limiting than potential differences among those included and excluded because of attrition. 


Table 3. Number of Principals From Selected Programs Included in the Analysis of 
Inexperienced Principals 


District 

Number of Principals Included in 
Inexperienced Principal Analysis 

Number of Selected Program Principals Dropped 
From Inexperienced Principal Analysis Because of 
Attrition 

A 

5 

0 

B 

5 

1 

C 

13 

0 

D 

21 

3 

E 

16 

1 
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Selection of Comparison Schools 

The disruption associated with principal turnover may influence student achievement in a school, 
regardless of how well or where a principal was trained. As described earlier, to account for this 
possibility, we included comparison schools in our models. These schools also received an 
inexperienced principal who was not trained by one of the selected programs. This is the first— 
and perhaps most important— criterion we used to determine the set of comparison schools: Only 
those schools that received a new principal who was not trained by one of the selected programs 
were included in the set of comparison schools. 

A second criterion relates to the baseline achievement and trends of the comparison schools. 
Comparison schools with baseline achievement levels and trends similar to those of schools that 
received an inexperienced principal from one of the selected programs may provide a better 
estimate of the counterfactual for treatment schools in the post period than one based on all schools 
that received an inexperienced principal. For example, the disruption associated with principal 
turnover may be greater in schools with lower baseline achievement levels. If that is the case, it is 
important that we base our estimate of the counterfactual on schools with lower baseline levels of 
achievement. We therefore also matched schools receiving principals from one of the selected 
programs to comparison schools with similar average test scores in the pretreatment period. 

To select comparison schools with similar average test scores in the pretreatment period, we used 
one of two approaches, depending on the district: Euclidean distance matching (Districts A, B, 
and C) or propensity score matching (District D). In the fifth district (District E), we were unable 
to identify comparison schools because we could not reliably identify when or where new 
principals from outside the selected program were placed. For the program serving that district, 
we modified our overall analytic strategy. 

In Districts A, B, and C, sometimes only one or two treatment schools were in each cohort. In 
these cohorts, little or no variation existed in the characteristics of schools receiving a principal 
from one of the selected programs. As a result, it is not possible to estimate propensity scores for 
these cohorts. Instead, we match based on the Euclidean distance between each treatment school's 
annual pretreatment average reading/ELA and mathematics test scores and comparison schools' 
annual pretreatment average reading/ELA and mathematics test scores . 8 To be consistent within 
districts, we used this Euclidean distance approach for every cohort in Districts A, B, and C, rather 
than matching some cohorts based on propensity scores and other cohorts in the same district 
based on Euclidean distance. In District A, where the number of potential comparison schools in 
each cohort was small, the treatment schools were matched to their two nearest comparison 
schools with replacement. In Districts B and C, the treatment schools were matched to their five 


8 For example, the distance between Comparison School 1 (cl) and Treatment School 1 (fl), both of which 
received an inexperienced principal in 2011-12, is calculated as follows: 

distance clM = JZsereading.math Sy=2°oo8-o9 [(tesTscore(s^y)^ - test score(s,y ) tl ) 2 ], 

where test score (s, y)c i is the average test score in subject s in school cl during year y, and test score (s, y)n 
is the average test score in subject s in school fl during year y. 
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nearest comparison schools, with replacement. In our analysis of the relative effectiveness of 
inexperienced principals from selected programs, comparison schools were weighted based on 
the number of times they were matched to a school that received an inexperienced principal from 
one of the selected programs. 

In District D, which has several principals from one of the selected programs in each cohort, we 
used propensity score matching. Propensity scores were estimated separately by cohort of 
inexperienced principals: graduates from selected programs whose first principal assignment 
was in 2011-12 were matched with others whose first principal assignment was in 2011-12, 
graduates from selected programs whose first principal assignment was in 2012-13 were matched 
with others whose first principal assignment was in 2012-13, and so on. Matches were based on 
the average ELA and mathematics scores of the school in the pretreatment period. For example, 
the propensity scores for schools receiving inexperienced principals in 2011-12 were based on six 
average scores for each school: average ELA scores in 2008-09, 2009-10, and 2010-11 and average 
mathematics scores in 2008-09, 2009-10, and 2010-11. Each treatment school was matched to the 
five nearest comparison schools, with replacement. In our analysis of the relative effectiveness of 
inexperienced principals from selected programs, comparison schools were weighted based on 
the number of times they were matched to a school that received an inexperienced principal from 
one of the selected programs. Details on the baseline test scores of treatment and comparison 
schools can be found in Appendix F. 

District E could not provide us with reliable information about principal assignments. In some 
years, the percentage of schools where District E data reported no principal was as high as 25%. 
However, we were able to obtain placement information from the preparation program itself. 9 
Our analysis for District E, therefore, did not include comparison schools and can be more 
accurately described as an interrupted time series rather than a CITS. Our model for District E 
described the change in student achievement following the arrival of an inexperienced principal, 
accounting for levels and trends in student achievement and changes in observable student 
characteristics at that school. 

We cannot necessarily attribute change in achievement, relative to expectations, to the principal. 
For example, change in achievement could be attributed to the impact of the principal, the 
disruption caused by principal turnover in the school, reversion to the mean of student 
achievement following the departure of a less-effective principal, or changes in districtwide 
school policies, among other factors. More details on the methodology used for District E can be 
found in Appendix F. 

Estimating Variation in Effectiveness Among Principals 

Although RQ3 focused on average effectiveness of a program's graduates, RQ3a asked about the 
variation in the effectiveness of program graduates in their first placements as principals. In 
addition to estimating average differences in effectiveness between inexperienced principals 


9 This program carefully tracks program graduates to provide ongoing coaching and support, and we 
therefore had confidence that the data about principal assignments were accurate. 
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trained by selected programs and other inexperienced principals in the same district, we also 
estimated the impact of each principal, which allowed us to quantify the variation of the 
effectiveness of inexperienced principals within each district. This analysis builds on the same 
samples of principals included in the previous analysis. 

To estimate the impact of individual principals on student achievement, we accounted for 
differences in levels and trends in student achievement between different cohorts of schools and 
between treatment and comparison schools within each cohort of schools. We also accounted for 
year-to-year changes in observable student characteristics in each school. We then determined 
how much higher or lower student test scores were at that school, on average, following the 
arrival of the inexperienced principal, compared with where one would have expected student 
test scores to have been. This determination was based on both previous levels and trends in 
student achievement in different groups of schools and changes in observable student 
characteristics within each school. The magnitude of this increase or decrease in average student 
achievement at a school following the arrival of the inexperienced principal, accounting for levels 
and trends in achievement and changes in student characteristics, was our estimate of the 
individual principal's contribution to student achievement. 

Estimating Effects of All Principals From Selected Programs 

Initial review of the program graduate data indicated that some program graduates already had 
experience as a principal either prior to or concurrent with attending the program. Other program 
graduates were placed as principals too early in the time series available for our analysis to meet 
the conditions necessary for the CITS (i.e., 3 years of pretreatment data for the schools they were 
placed in were not available). These individuals consequently did not meet the criteria of 
inexperienced and newly placed principals; excluding them from the analysis limited our ability 
to report on the overall effectiveness of the selected programs. 

To provide the most representative view of the program's graduates as possible, we added an 
additional research question (RQ4), which was aimed at examining the effects of all program 
graduates (regardless of experience). Table 4 shows the numbers of graduates included for each 
selected program. 


Table 4. Numbers of Principals From Selected Programs Included in Analysis 


District 

Number of Principals Included in 
"All Principals" Analysis 

Total Number of Graduates Serving as Principals 
for at Least 1 Year Between 2008-09 and 2014-15 a 

A 

11 

11 

B 

6 

6 

C 

33 

47 

D 

274 

382 

E 

— 

76 


“Graduates may be excluded if they did not serve as principal at least 6 months in a school year or if they served in a 
school without Grades 3-8 ELA or mathematics. 
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The goal of RQ4 was to determine whether students who attended schools led by principals 
trained by one of the selected programs had higher- or lower-than-expected levels of 
achievement, given their observable characteristics. 

To analyze the relative effectiveness of all (not just inexperienced) principals from selected 
programs, it was necessary to implement a different analytic strategy than the one used for RQ3. 
Recall that our CITS strategy grouped schools into cohorts that received principals with the same 
amount of experience (in this case, zero) in the same year. The analysis also required that several 
years of achievement data be available for those schools prior to the change in principals in that 
school. Those data were not available for all principals. Also, identifying comparison schools for 
all principals from one of the selected programs during the time period of our analysis that would 
match on relevant criteria, such as length of tenure in a school, total years of experience, 
achievement, and principal assignment, was not possible. We therefore adopted a different 
analytic strategy for this analysis. 

This strategy, a regression model with student and school fixed effects, followed the RAND 
(Gates et al., 2014) study of the relative effectiveness of principals trained by the New Leaders 
Program and serving as principals in 10 districts. The authors of that study noted that one might 
expect principal effects on student achievement to differ based on the number of years the student 
attends a school led by a treatment principal. For example, a student's achievement might 
increase in proportion to the number of years that the student attends a school led by a highly 
effective principal. The authors also noted that one might expect principal effects on student 
achievement to differ based on the length of the principal's tenure at the school. For example, if 
a highly effective principal has led a school for a number of years, that principal has had time to 
implement effective policies, hire effective teachers, provide coaching to less effective teachers, 
and improve the culture of the school, all of which could lead to an increase in student 
achievement. Like Gates et al. (2014), we therefore take two distinct approaches to defining 
students' treatment status: exposure and tenure. 

Exposure 

This model allows for the possibility that students with 1 year of attendance at a school led by a 
principal trained by one of the selected programs will be affected differently from students with 
2 years of attendance at a school led by a principal trained by one of the selected programs. 
Similarly, students with 2 years of attendance at a school led by a principal trained by one of the 
selected programs may be affected differently from students with 3 years of attendance at a school 
led by a principal trained by one of the selected programs. 

Tenure 

We expected that the longer a principal's tenure at a school, the more time the principal would 
have had to implement policies and other changes at the school. This model allowed principals' 
impact in their first year of tenure at a school to be different from their impact during their second 
year of tenure at a school, on average, and also allowed principals' impact in their second year of 
tenure at a school to be different from their impact during their third year or greater of tenure at 
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a school. Moreover, it allowed the impact of treatment principals in their first year of tenure at a 
school to be different from the impact of all other principals in their first year of tenure at a school; 
the impact of treatment principals in their second year of tenure at a school to be different from 
the impact of all other principals in their second year of tenure at a school; and the impact of 
treatment principals in their third year of tenure at a school to be different from the impact of all 
other principals in their third year of tenure at a school. 

The primary differences between the exposure and tenure models are as follows: 

• The exposure models assumed that the impact of principals from one of the selected 
programs was permanent. In contrast, the tenure models assumed principals affect students 
only when students were attending those principals' schools. As soon as the student 
changed to a school that was not led by a principal from one of the selected programs, or as 
soon as leadership at the school changed from a selected program principal to another 
principal, the impact of the selected program principal dissipated entirely. 

• The exposure models assumed that the impact of a principal from one of the selected 
programs increases in proportion to the number of years that a student attends a school 
led by a principal from one of the selected programs. The tenure models did not make this 
assumption. 

• The tenure models assumed that the impact of principals in their first year of tenure at a 
school was different from the impact of principals in their second year of tenure at a 
school, and the impact of principals in their second year of tenure at a school is different 
from the impact of principals with more than 2 years of tenure at a school. 

Moreover, the tenure models assumed that the impact of principals from one of 
the selected programs in their first year of tenure at a school was different from 
the impact of other principals in their first year of tenure at a school, the impact of 
principals from one of the selected programs in their second year of tenure at a 
school was different from the impact of other principals in their second year of 
tenure at a school, and the impact of principals from one of the selected programs 
with more than 2 years of tenure at a school was different from the impact of other 
principals with more than 2 years of tenure at a school. 

The exposure models did not account for principal tenure at the school. 

Accounting for Differences Between Students 

To account for the fact that students who attend schools led by treatment principals may have 
higher or lower baseline levels of achievement, on average, than the typical student in that 
district, we implemented a student fixed-effects model. In the exposure models, the student fixed- 
effects approach compared a student's achievement in years prior to attending a school led by a 
treatment principal with that student's achievement in years during or after which the student 
attended a school led by a treatment principal. In the tenure models, the student fixed-effects 
approach compared a student's achievement in years the student did not attend a school led by 
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a treatment principal with that student's achievement in years the student attended a school 
led by a treatment principal. 

Because these analyses depended critically on comparing a student's performance in treatment 
years with that student's performance in nontreatment years, the models included the following 
controls for time-varying individual student characteristics: 

• A binary "skip" variable, indicating the student was in a higher grade than one would 
expect given the student's first observed grade 

• A binary "retain" variable, indicating the student was in a lower grade than one would 
expect given the student's first observed grade 

• A binary "move" variable, indicating the student was attending a different school in the 
current year than the student attended the previous year 

• A vector of binary enrolled grade indicators 

• The "move" indicator interacted with enrolled grade indicators, to allow (for example) for 
the possibility that changing schools between Grades 5 and 6 might affect students 
differently from changing schools between Grades 6 and 7. 

• Binary variables indicating English language learner (ELL) status, students with 
disabilities (SWDs), and eligibility for participation in the free or reduced-price meal 
program in the current year 

Students' year-to-year movement between schools may explain important within-student 
changes in student achievement. For example, we might expect a dip in achievement among 
students who change schools between Grades 4 and 5, relative to students who do not change 
schools, because of the stresses related to changing schools. We identified the relative 
effectiveness of principals from selected programs based, in part, on student movement between 
schools led and not led by principals from selected programs. If students recently exposed to 
principals from selected programs also were students who had recently changed schools, failure 
to account this movement could bias our estimates of the relative effectiveness of principals from 
selected programs. 

Because our data sets cover only the years 2008-09 through 2014-15, we did not have accurate 
measures of student movement between 2007-08 and 2008-09. To ensure that we had accurate 
measures of student movement in all years covered by the analysis, we therefore dropped student 
test scores from 2008-09 from our analysis. This limited our analysis of the relative effectiveness 
of all principals to the years 2009-10 through 2014—15. 

Accounting for Differences Between Schools 

Our models accounted for the fact that students attending schools led by treatment principals 
may be different, on average, from other students in the district. Our models also accounted for 
the fact that the schools themselves might be different from the typical school in a district. To 
account for these differences, our model controlled for the characteristics of students at the school 
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each year, including the share of students who were female (where available), ELLs, SWDs, and 
eligible for free or reduced-price meals (where available), and the share of students in each of 
several large ethnicity categories. In addition, because schools where principals from selected 
programs were placed might be different in unobservable ways as well— the teachers and other 
staff may be different, the facilities may be different, and the surrounding community may be 
different— and these differences could affect student achievement, our models included time- 
invariant school fixed effects to allow for these potential differences. 

Accounting for Differences in Principal Experience 

Our models accounted for principal experience through a series of five binary variables (first year 
of experience, second year of experience, . . . , fifth year of experience, and an omitted category of 
six or more years of experience). Principal experience and principal tenure will differ for all 
principals who have led more than one school. The details of the exposure and tenure models can 
be found in Appendix F. 

Prior Research and Challenges 

Our analysis of the relative effectiveness of inexperienced principals trained by selected programs 
used a within-school design similar to that employed by Corcoran, Schwartz, and Weinstein (2012), 
although our study uses student-level, rather than aggregate school-level, data. Our analysis of the 
relative effectiveness of both experienced and inexperienced principals from selected programs 
followed directly from the strategy implemented by Gates et al. (2014). A number of challenges 
affect any research of this type. 

First, the process by which individuals are selected (or self-select) into preparation programs 
cannot be disentangled from estimates of program effectiveness. For example, some programs 
may be better at selecting individuals who become more effective principals (e.g., those with 
greater content knowledge or other relevant skills) than others. A program may then appear to 
be more effective because it selects highly qualified applicants, even though the program does 
little to improve the skills of those it prepares. However, the process by which a principal 
preparation program selects its applicants may be viewed as an important component of the 
program. The goal of this study was to determine the effect of principals from one of the selected 
programs and thus cannot explain why those principals were more effective. Follow-up studies 
of successful principals or programs with less variation might provide more insight into the 
specific practices of the programs that led to the observed outcomes. 

In some schools in our study, a principal trained by one of the selected programs was placed at a 
school, served for a number of years, left the school, and was replaced by another principal from 
one of the selected programs. This created a problem for our identification strategy for our models 
based on inexperienced principals: Should the first year the second principal leads the school be 
considered the first year the school is treated by an inexperienced principal from one of the 
selected programs or a continuation of the leadership from the previous principal trained by one 
of the selected programs? Because neither approach would provide a clean measure of the 
number of years the school was led by an inexperienced principal from one of the selected 
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programs, we dropped from our sample all observations where the school was led by a second 
principal from one of the selected programs. 

In some districts, many years pass between the time an individual completes one of the selected 
programs and the time the individual first becomes a principal. We did not control for the number 
of years that passed between the time an individual completed a principal preparation program 
and the time the individual became a principal. Individuals who did not secure a principal 
position until several years after completing their principal preparation program might be 
different, on average, from individuals who secured a principal position shortly after completing 
a principal preparation program. For example, individuals who did not secure a position until 
several years after completing the program might have been less effective than principals who 
secured a position immediately for reasons that had nothing to do with the span of time between 
completing the program and securing a position. Therefore, controlling for the number of years 
that passed between when an individual completed a preparation program and the time he or 
she secured a principal position could remove important variation in principal quality from our 
estimates of the relative effectiveness of the selected programs, and we therefore chose not to 
include such a control. 

In addition, as a practical matter, it would not be possible to compare principals from selected 
programs for whom a certain number of years passed between the completion of the selected 
program and their first principal assignment with principals from other programs for whom a 
similar number of years passed between the completion of their principal preparation program 
and their first principal assignment. We were not able to make such a comparison. Although we 
often had this information for principals trained from selected programs, we usually did not have 
this information for principals from other programs as a result of data quality issues. To the extent 
that the potential effect of the selected program decayed between the time the individual 
completed the program and the time the individual received his or her first principal assignment, 
our estimates of the relative effectiveness of principals from selected programs were attenuated. 
Finally, the time series of our analysis represents a 6-year period. To the extent that it may take 
principals several years to affect student achievement, it may be difficult to find effects for 
principals who assumed their leadership roles several years into this time period. 
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Findings 

Overview of Programs Selected for Study 

The programs selected for the study were based on the extent of their alignment to the selection 
criteria drawn from the best available literature on promising practices in preparing principals. 
The selected programs represent a broad range of both geographic location and program design. 
They include both traditional programs operated by large universities and large and small 
programs operated by nonprofit organizations. Some programs are run as a partnership between 
a university and a nonprofit provider. The programs vary in how long they have been in 
existence— with some being founded more than a decade ago and others still in their early years. 

Each selected program partners closely with one or more medium-to-large urban districts across 
the United States to place their graduates as leaders following graduation. 10 Some programs aim 
to place their graduates immediately into the principalship, whereas others place some graduates 
in an assistant principal or other leadership role, with the long-term goal of attaining a principal 
position. 

The program lengths range from 14 months to 5 years, with some programs resulting in a master's 
degree, a doctoral degree, or principal licensure. Several programs require their applicants to 
obtain their principal license or certification prior to enrolling in the program. 

All five selected programs include both graduate-level coursework and a minimum of one school 
year in an intensive clinical residency in one or more schools with support from a mentor 
principal. For some programs, the support for graduates continues after the graduate is placed as 
a principal (or other leader) in the partner district. We provide here a synthesis of how the five 
selected programs align to the selection criteria and more detailed specific examples of selected 
programs for each criterion. 

Although some of the five selected programs aim to place their graduates as principals within 
1 year of graduation from the program, others typically expect their graduates to go on to be 
assistant principals or serve in other leadership roles before becoming school principals. Four 
programs provided information for the rate at which their graduates are placed as principals 
within 1 year of graduation. Two programs provided additional placement rate information in 
alignment with the expected trajectory of their graduates into the placement role (Table 5). 


10 Although some of the selected programs placed graduates as principals in more than one district, this 
study only examined schools led by principals in one partner district for each program. 
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Table 5. Rate at Which Selected Programs Place Graduates Into the Principal Role 


District 

Number of Graduates as 

of 2013-14 

Percentage of Graduates 
Placed as Principals Within 

1 Year of Graduation 

Additional Placement 
Rates Provided by 
Program 

District A 

128 

12% 


District B 

35 

37% 


District C 

186 

N/A 

42% (within 3 years) 

52% (within 5 years) 

District D 

517 

80% 


District E 

148 

51% 

41% placed as assistant 
principals within 1 year 


Program Alignment to Research-Based Competencies 

The structure and curriculum of the five selected programs are organized according to a set of 
research-based standards and practices and display alignment to these research-based 
competencies. Most selected programs align their work to the Interstate School Leaders Licensure 
Consortium (ISLLC) standards, although they also may align to district-specific standards or 
competencies. Examples from two of the selected programs follow: 

• One program was codesigned by the university and the district. The internship and 
coursework is aligned to the district's leadership framework, which guides district 
assistant principal and principal evaluation. The framework is aligned to both the state 
principal standards and the ISLLC standards. The coursework focuses on developing 
specific skills, knowledge, and experiences related to specific expectations or 
competencies within the district leadership framework. Program faculty include 
university instructors and district leaders, and the internship is supported by practicing 
principals serving as mentors. This team uses the framework and several skills and 
competency-based rubrics to provide feedback and assess candidate performance on the 
projects and portfolios. Program faculty also assess participant performance on the 
leadership competencies through the district's assistant principal evaluation system. 
Program faculty meet with participants and mentor principals quarterly to discuss their 
performance and leadership competencies. 

• For another program, the competencies that drive the program most centrally are the 
district's principal competencies. These competencies are aligned closely to the ISLLC 
standards and were developed through collaboration among the district, the school, and 
other partners and reviewed by a district taskforce. Both the district's principal eligibility 
examination and principal evaluation system are aligned to the competencies. The program 
coursework also addresses a range of professional standards and content areas that are 
independent of district principal competencies, such as addressing the needs of ELLs. 
Throughout the program, participants receive continuous formative assessment of their 
school leadership practice as it aligns to the competencies. The program includes frequent 
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documentation and assessment of candidates on critical program components. For example, 
coaches rely on cumulative documentation of leadership learning through structured logs 
and school-based work products that illustrate progressive problem solving. 

Partnerships for Excellence 

In each selected program, district schools serve as sites for learning; in some cases, district 
personnel coteach classes and provide feedback on graduate quality and program curriculum. 
Each selected program has a strong partnership with the district(s) where graduates are placed 
as principals, although these partnerships can take different forms, such as the following: 

• One program was cofounded with district leadership as a collaborative effort, with district 
staff included on the program advisory board and in some cases serving as adjunct 
professors. As part of this partnership, following a candidate's residency placement in the 
district, district staff members evaluate the performance of the resident. This type of 
feedback loop continues after the program graduate is placed in the district as a principal; 
district staff members provide ongoing feedback on the graduates who are placed as 
principals in their schools. 

• Another program has a formal partnership with the district, where the district commits to 
recruit, select, and train a set number of principals to lead schools. As part of the contract, 
the district requires targeted recruitment of qualified district talent and alignment of the 
program curriculum to the district's principal evaluation rubric. In addition to these 
requirements, the program and the district have a data-sharing agreement, whereby data 
related to program graduate progress as measured by program assessments and related 
action plans are made available to the district, and district principal evaluation data of 
program graduates are shared with the program. Through this partnership, the program 
commits to conducting ongoing evaluation of graduates and providing quality mentors 
(i.e., the mentor principals at the residency schools). The selection of residency sites and 
mentor principals is done by the program but must be approved by the district. 

• Another program has signed memorandums of understanding (MOUs) with each partner 
district. The MOU stipulates shared responsibilities across the program and the partner 
districts during a 4-year period. District staff members are asked to encourage their best 
teacher leaders to apply to the program and publicize the program throughout the district. 
The MOU obligates districts to "provide dedicated and qualified" principals to serve as 
mentors to the program's residents, but it is up to the program to ultimately train these 
mentors. Both the district and the program are tasked with monitoring the residents' 
progress in fulfilling all the program's competencies. The MOU stipulates that the district 
should encourage key personnel to participate in program design and revision meetings. 
District leaders occasionally serve on panels for classes or give guest lectures, but they do 
not serve as full-time instructors to first-year students. 
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Experiential Learning 

Each selected program includes coursework and other student learning experiences that expose 
participants to the problems of school leadership practice, with the intent of building practical 
and technical knowledge. Although specific learning experiences vary across the programs, they 
are typically scaffolded, moving from classroom or online learning simulations to internship 
experiences, where students lead all or a significant portion of a school's operations. Here are 
some examples: 

• One program begins the curriculum sequence with candidates participating in a 6-week 
summer induction course, which serves as a simulation school. The purpose of this 
component of the program is to offer new participants a simulated leadership experience 
without the risks associated with a real school setting. This program develops an 
elementary and secondary simulation school along with detailed contextual information, 
operational documents (e.g., schedules and budgets), artifacts of teacher practice, such as 
videos of classroom instruction, profiles of teachers, and fictionalized student 
achievement data. Participants rotate through various leadership roles throughout the 
summer intensive course and engage in a series of high-stakes tasks. After completing the 
induction component of the program, program participants go through a comprehensive 
residency program at a host school, where they serve as an apprentice to an experienced 
mentor principal, which is an opportunity to gain field-based experience in a school 
similar to the one they will lead when completing the program. To support their work 
with the apprentices, mentor principals receive initial training and recurring visits by 
program facilitators, who collaborate with them to create learning experiences that meet 
the evolving needs of the program participant. The residency also includes regular check- 
ins with the mentor principal, program peers, and program staff facilitators. These 
facilitators engage in periodic observations of the participant to provide feedback, inform 
practice, and monitor the participant's progress toward goals. 

• One program includes a full-time paid internship that is integrated with inquiry-based 
projects. Interns work with their mentor principal and faculty to identify areas within the 
leadership framework of the partnership school district where they need additional 
experience. Mentors and cohort instructors work to provide on-the-job leadership 
experiences aligned with interns' needs to develop their leadership skills. This program's 
coursework includes four inquiry-based projects that are job-embedded with their 
internship. Each project is aligned to the leadership framework competencies and a 
problem of practice in a school setting. The project is customized to the intern's learning 
needs and the context and needs of the host school. For each project, interns collect, 
present, and analyze data through a portfolio website and receive feedback from peers, 
mentor principals, and cohort instructors. 

• Another program includes a 12-month paid residency during which the candidates have 
the opportunity to lead staff and build their own capacity as school leaders. After 
coursework in the winter and spring of their first year, each participant is matched with a 
mentor principal and residency school site in June and becomes a resident. The district 
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selects mentor principals with input from the program. Mentor principals build 
relationships and meet regularly with the resident and program coach, introduce the 
resident to teachers and other school staff, facilitate access to classroom observations to 
provide teacher feedback, include the resident in leadership team meetings, and provide 
opportunities for the resident to take on leadership roles. Each resident also has a program 
coach. Coaches are retired district principals deemed highly successful by the program. 
During the residency phase, coaches focus on the developmental needs of the resident in 
relation to the district's leadership competencies, residency goals, and action plan. 
Coaches conduct weekly 2-hour visits with the residency site and meet monthly with the 
mentor principal for a perspective on the resident's performance. In addition, the coach 
provides support and guidance to the resident regarding all interviews and placement 
offers that occur near the end of the residency. 

High-Quality, Rigorous Recruitment and Selection 

The selected programs engage in recruiting high-ability candidates with experience as expert, 
dynamic teachers and a commitment to instructional improvement. Some examples are as 
follows: 

• For one program, the university preparation provider and the district work in partnership 
to recruit and select participants for the program. The district has a process for collecting 
nominations that district staff members recommend for the program. Nominees who wish 
to be admitted must apply to the university master's program using traditional 
admissions procedures. Eligible applicants then participate in an in-person, three-part 
screening process: a writing prompt, a role-playing activity in which the applicant offers 
feedback to a teacher based on a video of the teacher's instruction, and a prioritization 
activity. An interview with university and district personnel is then held, and all applicant 
responses are rated on a rubric. The rubric measures several district principal 
competencies, including how the applicant works with people and what his or her basic 
human capital management and time management skills are. Faculty members then 
conduct a telephone interview with the applicant's referring principal, after which the 
program develops a list of the top 20-25 candidates. This final list of candidates is shared 
with district regional superintendents who review the selected participants and inform 
the applicants in writing of the program admission decision. 

• Another program has a multistage recruitment process. The process consists of 
nomination by a partner district, application to the program, a telephone interview, and 
an in-person exercise at the program's assessment center. Participants commit to the 
program for 5 years, and they must meet all the entry requirements for the graduate 
school where the program is housed. After candidates submit a written application, the 
next step is a 2-hour, two-part telephone interview. Immediately before the interview, 
candidates receive a case study and have 1 hour to examine it and think about how they 
would handle the described situation. Applicants spend the next half hour discussing the 
case with the interviewer, and the interviewer scores the candidates' responses using a 
rubric based on the program's core competencies. The final half hour consists of a more 
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traditional job interview. Candidates who pass the telephone interview stage are 
recommended to the assessment center for in-person activities, which includes an 
intensive interview and appraisal that lasts approximately 4 hours. Candidates go 
through a series of activities that simulate the experiences that a new principal may 
encounter on the job. For instance, candidates might role-play scenarios in which they 
interact with a student, a teacher, and a parent. They also are asked to observe video 
recordings of teachers, evaluate the instruction, and then provide feedback in a role- 
playing scenario. Program staff members evaluate candidates' instructional knowledge 
and their ability to communicate and have difficult conversations with staff. Finally, there 
is an exit interview, where candidates are given tough feedback to assess how they deal 
with criticism. These exercises are assessed using a rubric that aligns with the program's 
competencies for transformational leadership. Some candidates are offered admission to 
the program immediately after the in-person exercises; others may be required to 
complete one final interview to learn more about their qualifications before a final 
decision is reached. Approximately 30% to 40% of candidates who participate in the 
assessment center exercise are offered admittance into the program. 

• The recruitment and selection process for a third program involves the university's 
admissions office and faculty, program graduates, and district and school leaders. The 
process includes general information sessions, meetings with superintendents and 
principals on the benefits of the program for the school and district, communication with 
program graduates about nominating prospective candidates, and conversations with 
principals and district leaders about opportunities to partner with the program. Alumni 
referrals are the most common recruitment effort, and graduates are encouraged to 
nominate potential candidates. Graduates also typically lead recruitment events by sharing 
their experiences and answering questions. The program recruits primarily from within the 
partner district but also recruits nationally through a partnership with an alternative 
teaching certification program. Applicants must demonstrate effectiveness as a teacher or 
comparable effectiveness in another role within the district. The program receives 
approximately 50 applications per year for 16 open spots. The application package requires 
a personal essay, a transcript, a resume, a copy of the teacher's license, and two nominations 
from school or district leadership. The university and district staff partner to review each 
application and rate applicants based on their leadership and teaching experience, 
commitment to lead change toward high achievement and equity, and evidence of a 
learning and growth mind-set. Applicants who survive this first screening then participate 
in a 2-hour interview that includes scenarios or activities based on competencies from the 
district's leadership framework (which is used to evaluate sitting assistant principals and 
principals). A panel of university faculty, program graduates, and district administrators 
score each activity on a competency-based rubric that is aligned to the district's standards 
for principals. The final selection of candidates depends on the needs of the district, 
including a prioritization of the diversity of the cohort. 
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Early Years on-the-job Support 

The selected programs include intensive on-the-job support (i.e., induction) for program 
graduates who are principals, such as mentoring, coaching, or a peer network for at least 1 year 
after graduation. Some examples are as follows: 

• Until recently, when grant funding provided for an expansion of coaching support, one 
program provided 1 year of coaching support to graduates in their first year as principals, 
with about 72 hours of support funded by the district during the course of the school year. 
After the first year of district-funded coaching support, principals had the option of 
purchasing additional coaching support in subsequent years from their school budget. 
Program coaches receive extensive professional development, including support for 
contextual needs (e.g., district initiatives, accountability measures, and the use of data) 
and best practices in coaching support. Throughout their tenure, coaches receive feedback 
to improve their practice, observations in the field by program staff, and both large-group 
and individual professional development. Coaches create a professional development 
plan aligned to their individual goals. This program recently expanded its coaching 
support with grant funding to include up to 164 coaching hours for first-year principals, 
including a diagnostic assessment of school and leader needs, visits to other schools led 
by program graduates, and annual convenings across schools for reviewing and sharing 
accomplishments. Coaches who are part of this expanded model receive training in team 
coaching, as well as the school diagnostic process, in addition to the training provided to 
all coaches described earlier. 

• Another program blends theory and practice during and after the residency. The preservice 
part of the program concludes 18 months after matriculation, when all state requirements 
are met and the yearlong residency has been completed. In the final months of their 
residency year, participants actively seek out in-service leadership positions, ranging from 
principal to assistant principal to district-level administrator, and they begin to receive in- 
service supports provided by the program. The program believes that guided reflective 
practice through coaching holds candidates accountable for continuous long-term growth 
and is critical for producing truly transformational instructional leaders. The school 
leadership coaching that begins during the residency continues when each candidate 
assumes a position as a newly licensed principal, as an assistant principal, or in a district- 
level leadership role. Coaches continue to work with candidates who were initially assigned 
to them for up to 3 years after the residency. The frequency of on-site coaching support for 
novice principals depends on the needs of the new principal as determined by the coach. 

• One program provides participants with several layers of support from the program 
during their initial years of leadership in the form of coaching and mentorship. 
Participants first receive mentoring during their residency year (Year 2 of the program), 
where they are paired with a mentor principal. Mentor principals must apply to the 
program and demonstrate strong achievement and values that align with the program's 
leadership competencies. The mentor may or may not be the resident's host principal. 
Mentors are tasked with creating learning opportunities, supporting residents' action- 
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learning projects, providing feedback along the program's core competencies, and 
allowing residents to sit in on high-level school and district meetings. The program 
provides mentors with a series of professional development sessions. Mentors also assist 
in the recruitment process by evaluating candidates in the assessment center. Mentors 
meeting all the criteria for the year are paid a small stipend. The program also employs a 
team of former principals or experienced school leaders as leadership development 
coaches to support residents and recent graduates. These coaches are expected to check in 
with an assigned cadre of residents for two 2-hour sessions per month to provide feedback 
on residents' progress as school leaders. The coaches also observe residents in their 
schools, and they meet with the residents' host and mentor principals every other month 
to assess their progress toward meeting the program's competencies. The coaches provide 
similar but less intensive supports to recent graduates after their residency based on need 
and the participant's role in school leadership. 

Evidence of Effects 

The selected programs seek out evidence of effects on advancing leadership practice, school 
culture, or student learning that have been developed through rigorous study. In two states 
where the selected programs are located, law and administrative rules require that the programs 
collect graduate placement and performance data. The selected programs have established 
routines for collecting program effectiveness information and use the data to adjust program 
design, including the following: 

• For one program, after graduates have been in a principal position for at least a year, the 
program solicits an evaluation of the graduate from his or her supervisor in the district. The 
program also solicits district feedback on program participants after their internships. The 
program analyzes publicly available data from the district, including teacher surveys and 
working conditions surveys, to gauge graduate effectiveness and potential changes to the 
program's curriculum. The program has an advisory board of graduates who meet twice 
per year. Findings from all data collected are shared with the board, and the board provides 
recommendations as to what might be added, changed, or deleted in the current program. 

• One program has adjusted its curriculum and program components to adapt to emerging 
needs in the district. The program has a quarterly meeting with faculty instructors to discuss 
recommendations and areas for improvement in the curriculum. Instructors have adjusted 
the criteria and requirements of the inquiry-based projects to better support the 
development of participants' leadership skills based on the district's leadership framework. 
Graduates of the program also provide feedback and input to the program through a 
professional learning community consisting of graduates who are currently serving as 
principals. The program is evaluated both internally and externally. Internally, program 
faculty collect both formative and summative feedback. Program staff members examine 
data such as graduate placement and the performance of schools led by graduates. Program 
staff members conduct surveys of their graduates, asking their perceptions of the program's 
effectiveness in preparing interns for each competency. Program interns also are evaluated 
by the district's assistant principal evaluation system during their internship. 
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• One program annually presents progress on outcomes to their board, in addition to 
engaging in multiple types of internal and external data collection and evaluation. The 
outcomes the program tracks and reports include cohort diversity, graduation rates, rate of 
placement in the first year after graduation, retention beyond the first year, tenure rates, 
retention beyond 5 years, cohort satisfaction with coaching services, and the percentage of 
program principals who stay involved with the program in various capacities (e.g., as 
mentors, coaches, or participants). In 2013, the program reported meeting or exceeding 
nearly all its success metrics. For example, the program exceeded its recruitment diversity 
and graduation rate targets and met its principal placement rate target. The program also 
surveys its graduates each year on their own perceptions of the program and how the 
program prepared them to lead a school. These surveys have found that the majority of 
graduates are satisfied with the preparation they received through the program. The 
program also collects principal effectiveness data compiled by the district to monitor the 
effectiveness of its graduates, as measured by school progress reports and student 
achievement data for the schools their graduates lead. Finally, this program has engaged in 
several external evaluations and research studies, including a study examining the impact 
of graduates on student achievement. The program also is currently undergoing a rigorous 
evaluation study to evaluate the program's effectiveness in its stated goals for coaching, 
support, and retention of graduates as principals in district schools, as well as the effects of 
principal and assistant principal partnerships on student achievement, school climate, 
retention, and teacher satisfaction in the schools they lead. 

Principal Perceptions of Programs 

The results of the 68 interviews conducted with principals who graduated from the five selected 
programs included generally positive feedback on the programs. The goal of these interviews 
was to gather additional information about the practices employed by selected programs. 
Principal respondents described program design and program supports that were aligned both 
to the selection criteria used to identify programs for participation in the study and how the 
programs described themselves through program documentation and interviews. Key findings 
from these interviews are described here. The detailed findings from these interviews are in 
Appendix E. 

• All principals interviewed indicated that the selected principal preparation program they 
attended was aligned to a specific set of research-based principal standards or 
competencies. When asked to what extent, the vast majority (90%) said that their program 
was aligned to a set standards to "a great extent," and the remaining 10% of the 
respondents said to "a moderate extent." 

• Approximately 75% of the respondents found the coursework to be supportive of their 
practice in the principal role to "a great extent," with the majority also reporting that they 
applied their learning in the selected principal preparation program to their work in the 
principal role. 


35 


• The respondents generally reported favorable perceptions of the residencies and 
internships they experienced during their preparation, noting the value of these hands-on 
experiences in preparing them for the principal role. In one program, participants were 
afforded two types of internship experience and may have had issues with one, but overall 
favored the internship or residency experiences. 

• Ninety-nine percent of the respondents described multiple components and phases of the 
recruitment process (e.g., impromptu essays and scenario role-plays). Only one respondent 
described a straightforward interview process. In providing descriptions of the selection 
process, 9% of the respondents used the word rigorous, 6% of the respondents used the 
word intense, and 1% of the respondents described the selection process as extensive. 

• Across all the selected principal preparation programs, the respondents described varied 
levels of support they received once they were in the principal role. Respondents from 
some selected programs noted that support for sitting principals was primarily offered 
through the district, with the respondents from one district describing a multiyear 
comprehensive support system. The respondents from other selected programs continued 
to receive coaching support from the program once they were in the principal role. They 
also described more informal supports that they received through colleagues and other 
leaders in the district. 

• Approximately half of the respondents (51% of the 41 respondents who answered this 
question) whose programs provided coaching support to new principals found the 
coaching support they received from their program to be "greatly beneficial" to their 
practice in the principal role. Other respondents described the coaching provided as 
"somewhat beneficial" (37%) or only "beneficial" (7%). 

• When asked about the most useful aspects of the program, the respondents had many 
favorable perceptions to share, noting that the most useful aspects of the selected principal 
preparation programs were as follows: 

Internship or residency 

Mentoring 

Coaching 

Focus on instructional leadership 
Reflections on the realities of the job of principal 
Cohort model and networking 
Role-playing and simulation exercises 

• The respondents reported variability in the district support they received once they 
assumed the role of principal. For example, one district had a structured program of 
support, whereas other districts offered optional support components or components that 
could be purchased after a provision period that was free of charge. 

• The respondents also had mixed perceptions of the usefulness of the district support, with 
some respondents noting very minimal or ineffective district support, and others praising 
extensive support from the district. Some respondents also noted a lack of differentiation 
in district support. 
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Effects of Newly Placed (Inexperienced) Principals 
on Student Achievement 

In this section, we provide the results of the CITS analysis of principals in Districts A, B, C, and 
D. First, we describe demographic information for each district. Following this descriptive 
information, we present the results of the CITS analysis. Detailed findings (including some of the 
tables referenced in the body of the report) are in Appendix G. 

The results for District E are described separately in a subsequent section because of differences 
in the analytic strategy used to estimate the results. 

Students and Schools Included in the Analysis of Inexperienced Principals 

District A 

Table 6 shows the numbers of schools and principals included in analysis of inexperienced 
principals (i.e., schools in District A that received newly trained principals in 2011-12, 2012-13, 
or 2013-14). 

Table 6. Number of Schools in the Analysis of Relative Impact of Inexperienced Principals 
From One of the Selected Programs for District A 


Year 

Cohort 2012 

Cohort 2013 

Cohort 2014 

Cohort 2015 

Total Number 

of Schools 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

4 

0 

3 

0 

4 

0 

0 

0 

11 

0 

2009-10 

4 

0 

3 

0 

4 

0 

0 

0 

11 

0 

2010-11 

4 

0 

3 

0 

4 

0 

0 

0 

11 

0 

2011-12 

2 

2 

3 

0 

4 

0 

0 

0 

9 

2 

2012-13 

2 

2 

2 

1 

4 

0 

0 

0 

8 

3 

2013-14 

2 

2 

2 

1 

2 

2 

0 

0 

6 

5 

2014-15 

2 

2 

2 

1 

2 

2 

0 

0 

6 

5 


Altogether, 11 schools included in our study received inexperienced principals in one of the 
3 years, and five of the schools had principals who were trained by the selected program (Table 6). 
Only Grades 3-5 students were included in the models for District A (Table G-5) because only 
elementary schools received an inexperienced principal from one of the selected programs during 
this period. 

Schools led by a principal from one of the selected programs had larger percentages of Asian- 
American and White (Table G-3) students and smaller percentages of African-American students. 
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Hispanic students (Table G-3), ELLs, and SWDs (Table G-4). Information on students' eligibility 
for free or reduced-price meals was not available for District A. 

Detailed information on how long principals served in schools from 2011-12 to 2014-15 and the 
numbers and demographics of students in schools can be found in Appendix G, Table G-l 
through Table G-5. 

District B 

Table 7 shows the numbers of schools and principals included in the analysis of inexperienced 
principals (i.e., schools in District B that received newly trained principals in 2011-12, 2012-13, or 
2013-14). 


Table 7. Number of Schools in the Analysis of the Relative Impact of Inexperienced 
Principals From One of the Selected Programs for District B 


Year 

Cohort 2012 

Cohort 2013 

Cohort 2014 

Cohort 2015 

Total Number 

of Schools 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2009-10 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2010-11 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2011-12 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2012-13 

0 

0 

5 

1 

9 

0 

7 

0 

21 

1 

2013-14 

0 

0 

5 

1 

7 

2 

7 

0 

19 

3 

2014-15 

0 

0 

5 

1 

7 

2 

5 

2 

17 

5 


Altogether, 22 schools included in our study received inexperienced principals in one of the 
3 years, and five of the principals were trained by one of the selected programs. Only Grades 3-5 
students were included in the analysis for District B (Table G-ll) because only elementary schools 
received an inexperienced principal from one of the selected programs during this period. 

Among schools in District B where an inexperienced principal was placed in 2012-13, 2013-14, or 
2014-15, schools led by a principal from one of the selected programs had larger percentages of 
White students (Table G-9), ELLs, and SWDs (Table G-10), and these schools had smaller 
percentages of Asian- American (Table G-9) students. 

Detailed information on how long principals served in schools from 2011-12 to 2014-15 and the 
numbers and demographics of students in schools can be found in Appendix G, Table G-6 
through Table G-ll. 
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District C 

Table 8 shows the numbers of schools and principals included in the analysis of inexperienced 
principals (i.e., schools in District C that received newly trained principals in 2011-12, 2012-13, 
or 2013-14). 

Table 8. Number of Schools in the Analysis of the Relative Impact of Inexperienced 
Principals From One of the Selected Programs for District C 


Year 

Cohort 2012 

Cohort 2013 

Cohort 2014 

Cohort 2015 

Total Number 

of Schools 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

8 

0 

11 

0 

13 

0 

9 

0 

41 

0 

2009-10 

8 

0 

11 

0 

13 

0 

9 

0 

41 

0 

2010-11 

8 

0 

11 

0 

13 

0 

9 

0 

41 

0 

2011-12 

5 

3 

11 

0 

13 

0 

9 

0 

38 

3 

2012-13 

5 

3 

7 

4 

13 

0 

9 

0 

34 

7 

2013-14 

5 

3 

7 

4 

9 

4 

9 

0 

30 

11 

2014-15 

5 

3 

7 

4 

9 

4 

7 

2 

28 

13 


Altogether, 41 schools included in our study received inexperienced principals in one of the 
3 years, and 13 of the principals were trained by one of the selected programs. 

Among schools in District C where an inexperienced principal was placed in 2011-12, 2012-13, 
2013-14, or 2014-15, schools led by a principal from one of the selected programs had large 
percentages of Hispanic students (Table G-15), ELLs, SWDs, students from low-income families 
(Table G-16), and Grades 3-5 (Table G-17) students, and these schools had smaller percentages of 
African-American, Asian- American, White (Table G-15), and Grades 6-8 (Table G-17) students. 

Detailed information on how long principals served in schools from 2011-12 to 2014-15 and the 
numbers and demographics of students in schools can be found in Appendix G, Table G-12 
through Table G-17. 

District D 

Table 9 shows the numbers of schools and principals included in the analysis of inexperienced 
principals (i.e., schools in District D that received newly trained principals in 2011-12, 2012-13, or 
2013-14). We were not provided placement information for principals from one of the selected 
programs for 2014—15. 
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Table 9. Number of Schools in the Analysis of the Relative Impact of Inexperienced 
Principals From One of the Selected Programs for District D 


Year 

Cohort 2012 

Cohort 2013 

Cohort 2014 

Cohort 2015 

Total Number 

of Schools 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

27 

0 

30 

0 

18 

0 


75 

0 

2009-10 

27 

0 

30 

0 

18 

0 

75 

0 

2010-11 

27 

0 

30 

0 

18 

0 

75 

0 

2011-12 

21 

6 

30 

0 

18 

0 

69 

6 

2012-13 

21 

6 

23 

7 

18 

0 

62 

13 

2013-14 

21 

6 

23 

7 

10 

8 

54 

21 

2014-15 

21 

6 

23 

7 

10 

8 

54 

21 


Altogether, 75 schools included in our study received inexperienced principals in one of the 
3 years, and 21 of the principals were trained by one of the selected programs. 

Among schools in District D where an inexperienced principal was placed in 2011-12, 2012-13, or 
2013-14, schools led by a principal from one of the selected programs had larger percentages of 
Asian- American students, African-American students (Table G-21), ELLs, students from low- 
income families (Table G-22), and Grades 3-5 (Table G-23) students; these schools had smaller 
percentages of Hispanic students. White students (Table G-21), SWDs, and Grades 6-8 (Table G-23) 
students. 

Detailed information on how long principals served in schools from 2011-12 to 2014—15 and the 
numbers and demographics of students in schools can be found in Table G-18 through Table G-23 
in Appendix G. 

Relative Effectiveness of Inexperienced Principals From Selected Programs 

Table 10 and Table 11 describe the relative effectiveness of principals from selected programs 
across Districts A, B, C, and D. The findings for District E are presented separately. We describe 
the findings from each district in more detail following these tables. The detailed findings are in 
Appendix G. 

Across all four districts, we found little evidence that inexperienced principals from selected 
programs were more or less effective at fostering student achievement in mathematics. In one of 
the four districts (District A), we found that reading/ELA achievement is lower than expected in 
schools that received an inexperienced principal from one of the selected programs. This 
difference was driven primarily by newer cohorts of principals who had shorter tenures in their 
schools; as such, this may not reflect the long-term relative effectiveness of principals from 
selected programs. In Districts B, C, and D, we found no evidence that inexperienced principals 
from selected programs were more or less effective than other inexperienced principals. 
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Table 10. Relative Effectiveness of Inexperienced Principals From Selected Programs 



Covariate 

District A 

District B 

District C 

District D 


Reading 

Mathematics 

Reading 

Mathematics 

Reading 

Mathematics 

ELA 

Mathematics 

Treatment * Post 

-0.077** 

-0.059 

0.083 

-0.070 

0.029 

0.020 

-0.048 

-0.021 

(0.038) 

(0.057) 

(0.091) 

(0.102) 

(0.030) 

(0.040) 

(0.026) 

(0.032) 

Post 

0.025 

0.005 

0.033 

-0.121* 

-0.031* 

-0.032* 

-0.030* 

-0.052** 

(0.025) 

(0.040) 

(0.031) 

(0.051) 

(0.017) 

(0.017) 

(0.014) 

(0.016) 

Students 

24,382 

24,457 

20,661 

33,413 

64,906 

66,406 

191,627 

194,635 

R-Squared 

0.32 

0.32 

0.09 

0.09 

0.42 

0.38 

0.37 

0.39 


Robust standard errors, clustered within school-by-year cell, are in parentheses. 

**p<0.01.*p <0.05. 

See Table F-l in Appendix F for a complete list of model covariates. 

Table 11. Relative Effectiveness of Principals From Selected Programs, by Cohort 



Covariate 

District A 

District B 

District C 

District D 



Reading 

Mathematics 

Reading 

Mathematics 

Reading 

Mathematics 

ELA 

Mathematics 


0.122 

0.010 



0.059 

0.071 

-0.029 

0.010 

(eonort - zuii-izj ireatment lost 

(0.072) 

(0.073) 



(0.064) 

(0.102) 

(0.048) 

(0.056) 


-0.137** 

-0.156** 



0.010 

-0.030 

-0.039 

-0.044 

(v^onort - zUll-lzj l ost 

(0.048) 

(0.048) 



(0.046) 

(0.058) 

(0.042) 

(0.065) 


-0.161 

0.180 

0.246 

-0.396* 

0.034 

0.017 

-0.077* 

-0.034 

(eonort - zuiz-u) treatment lost 

(0.085) 

(0.108) 

(0.174) 

(0.165) 

(0.060) 

(0.073) 

(0.037) 

(0.048) 


0.121* 

-0.115 

-0.002 

-0.016 

-0.024 

-0.027 

-0.010 

-0.049 

(L-onort - zuiz-lo) l ost 

(0.057) 

(0.073) 

(0.053) 

(0.074) 

(0.025) 

(0.029) 

(0.022) 

(0.029) 


-0.267** 

-0.196 

- 0.111 

-0.314* 

0.041 

0.104 

0.005 

-0.069** 

(eonort - zu u-14 j ireatment lost 

(0.074) 

(0.099) 

(0.107) 

(0.121) 

(0.042) 

(0.059) 

(0.027) 

(0.025) 
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Covariate 

District A 

District B 

District C 

District D 

Reading 

Mathematics 

Reading 

Mathematics 

Reading 

Mathematics 

ELA 

Mathematics 

(Cohort = 2013-14)* Post 

0.177** 

0.092 

0.066 

-0.115 

0.002 

-0.044 

-0.073** 

-0.039 

(0.060) 

(0.061) 

(0.074) 

(0.062) 

(0.026) 

(0.037) 

(0.020) 

(0.031) 

(Cohort - 2014-15) * Treatment * Post 



0.361** 

0.213 

-0.077 

-0.139 





(0.133) 

(0.133) 

(0.068) 

(0.097) 



(Cohort - 2014-15)* Post 



0.016 

-0.197* 

-0.115** 

-0.043 





(0.038) 

(0.091) 

(0.038) 

(0.031) 



Students 

24,382 

24,457 

20,661 

33,413 

64,906 

66,406 

191,627 

194,635 

R-Squared 

0.32 

0.32 

0.10 

0.09 

0.41 

0.36 

0.37 

0.39 


Robust standard errors, clustered within school-by-year cell, are in parentheses. 
**p< 0.01. ><0.05. 

See Table F-l in Appendix F for a complete list of model covariates. 


District A 

In District A, following the arrival of a new inexperienced principal from the selected program, 
achievement was, on average, 0.08 SD lower than expected in reading and 0.06 SD lower than 
expected in mathematics. Only the result for reading was statistically significant. (See Figure 3 
and the first two columns of Table 10.) These average differences between principals from one of 
the selected programs and other inexperienced principals in District A were driven primarily by 
principals who were placed in 2013-14. In schools where inexperienced principals from one of 
the selected programs were placed in 2013-14, student achievement was 0.27 SD lower than 
expected in reading and 0.20 SD lower than expected in mathematics relative to changes in 
achievement in the comparison schools, although only the result for reading was statistically 
significant. (See Figure 4 and the first two columns of Table 11.) These results are based on only 
five principals from one of the selected programs in District A; as such, the results may not be 
representative of all principals from one of the selected programs in that district. 


Figure 3. Change in Student Reading Achievement Following the Placement of an 
Inexperienced Principal, District A 



Note. SPPP = selected principal preparation program. 
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Figure 4. Change in Student Mathematics Achievement Following the Placement of an 
Inexperienced Principal, District A 



Note. SPPP = selected principal preparation program. 

District B 


In schools where inexperienced principals from one of the selected programs were placed in 
District B, achievement was 0.08 SD higher than expected in reading and 0.07 SD lower than 
expected in mathematics relative to other schools in the district where inexperienced principals 
were placed, but neither difference was statistically significant. (See Figure 5, Figure 6, and the 
third and fourth columns of Table 10.) However, as the third and fourth columns of Table 11 
show, the relative effectiveness of principals from one of the selected programs differed across 
cohorts. In schools where an inexperienced principal from one of the selected programs was 
placed in 2014-15, student achievement in reading was 0.36 SD higher than expected, and the 
difference was statistically significant. But in schools where an inexperienced principal from one 
of the selected programs was placed in 2012-13, student achievement in mathematics was about 
0.40 SD lower than expected, whereas in schools where an inexperienced principal from one of 
the selected programs was placed in 2013-14, student achievement was 0.31 SD lower than 
expected. As with District A, however, the results for District B are based on only five principals 
from one of the selected programs. 
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Figure 5. Change in Student Reading Achievement Following the Placement of an 
Inexperienced Principal, District B 



Note. SPPP = selected principal preparation program. 
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Figure 6. Change in Student Mathematics Achievement Following the Placement of an 
Inexperienced Principal, District B 



Note. SPPP = selected principal preparation program. 

District C 


In schools where inexperienced principals from one of the selected programs were placed in 
District C, reading achievement was 0.03 SD higher than expected, whereas mathematics 
achievement was 0.02 SD higher than expected, but neither difference was statistically significant. 
(See Figure 7, Figure 8, and the fifth and sixth columns of Table 10.) In schools where 
inexperienced principals from one of the selected programs were placed between 2011-12 and 
2013-14, student achievement was higher than expected in both mathematics and reading, but 
neither difference was statistically significant. (See the fifth and sixth columns of Table 11.) In 
schools where inexperienced principals from one of the selected programs were placed in 2014- 
15, student achievement in both mathematics and reading were lower than expected, but the 
differences were not statistically significant. 
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Figure 7. Change in Student Reading Achievement Following the Placement of an 
Inexperienced Principal, District C 



Note. SPPP = selected principal preparation program. 
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Figure 8. Change in Student Mathematics Achievement Following the Placement of an 
Inexperienced Principal, District C 



Note. SPPP = selected principal preparation program. 

District D 


In schools where inexperienced principals from one of the selected programs were placed in 
District D, achievement was 0.05 SD lower than expected in ELA and 0.02 SD lower than expected 
in mathematics relative to other schools where inexperienced principals were placed, but neither 
result was statistically significant. (See Figure 9, Figure 10, and the sixth and seventh columns of 
Table 10.) We observed some statistically significant differences within cohorts. In schools where 
inexperienced principals from one of the selected programs were placed in 2012-13, student ELA 
achievement was 0.08 SD lower than expected, on average, and the difference was statistically 
significant. In schools where inexperienced principals from one of the selected programs were 
placed in 2012-13, student mathematics achievement was 0.07 SD lower than expected, on 
average, and the difference was statistically significant. (See the seventh and eighth columns of 
Table 11.) 
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Figure 9. Change in Student Reading Achievement Following the Placement of an 
Inexperienced Principal, District D 



Note. SPPP = selected principal preparation program. 
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Figure 10. Change in Student Mathematics Achievement Following the Placement of an 
Inexperienced Principal, District D 



Note. SPPP = selected principal preparation program. 


Changes in Student Achievement (District E) 

We performed a separate analysis for District E, based on the limited availability of principal 
placement data from the district (described in more detail in the Methods section of this report). 
Here, we describe demographic information about District E, followed by the results of our 
analysis. 

Students and Schools Included in the Analysis of District E 

Sixteen schools were included in our analysis of District E: Four schools received an 
inexperienced principal from one of the selected programs in 2011-12, three schools received 
an inexperienced principal from one of the selected programs in 2012-13, two schools 
received an inexperienced principal from one of the selected programs in 2013-14, and seven 
schools received an inexperienced principal from one of the selected programs in 2014-15. 
Additional findings are in Appendix G. 
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Changes in Student Achievement Following the Arrival of a Principal From One of 
the Selected Programs in District E 

Following the arrival of an inexperienced principal from one of the selected programs in District 
E, student reading achievement was about 0.06 SD lower than would otherwise be expected 
(Table 12), but the results were not statistically significantly different from zero (i.e., we cannot 
reject the possibility that there was no change). In mathematics, student achievement was about 
0.04 SD lower than would otherwise be expected, but again the results were not statistically 
significant. 

Looking at the changes in student achievement separately by cohort of inexperienced principals, 
we saw that most of the drop in student achievement can be attributed to inexperienced principals 
who were placed in 2014-15, with a lesser extent by inexperienced principals who were placed in 
2012-13. In schools where inexperienced principals from one of the selected programs were 
placed in 2014—15, student reading achievement was about 0.19 SD lower than would otherwise 
be expected, given prior levels and trends in student achievement in those schools and changes 
in the characteristics of students in the schools. Similarly, in schools where inexperienced 
principals from one of the selected programs were placed in 2014-15, student mathematics 
achievement was about 0.15 SD lower than would otherwise be expected, given prior levels and 
trends in student achievement in those schools and changes in the characteristics of students in 
schools. Similar drops occurred in reading and mathematics following the arrival of an 
inexperienced principal in 2012-13, but one cannot reject the null hypothesis that no change 
occurred following the placement of inexperienced principals from one of the selected programs. 

It is possible that student reading and mathematics achievement in District E schools where 
inexperienced principals were placed in 2014-15 will eventually return to their previous levels, 
as appears to be the case for the other cohorts. It also is possible that the drop in achievement 
experienced by these schools is unrelated to the arrival of inexperienced principals in one of the 
selected programs. 


Table 12. Change in Student Achievement Following the Arrival of a Principal From One of 
the Selected Programs in District E 


Covariate 

Reading 

Mathematics 

Model 1 

Model 2 

Model 1 

Model 2 

Post 


-0.059 


-0.043 


(0.035) 


(0.043) 

(Cohort = 2011-12) * Post 

0.089 


0.074 


(0.058) 


(0.060) 


(Cohort - 2012-13)* Post 

-0.081 


-0.109 


(0.077) 


(0.079) 


(Cohort - 2013-14)* Post 

0.120 


0.166 


(0.080) 


(0.121) 
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Covariate 

Reading 

Mathematics 

Model 1 

Model 2 

Model 1 

Model 2 

(Cohort - 2014-15)* Post 

-0.194** 


-0.152* 


(0.045) 


(0.071) 


Students 

37,262 

37,262 

37,437 

37,437 

R-Squared 

0.464 

0.464 

0.447 

0.447 


Robust standard errors, clustered within school-by-year cell, in parentheses. 
**p<0.01.*p <0.05. 

See Table F-l in Appendix F for a complete list of model covariates. 


Variation in Effectiveness Among Newly Placed (Inexperienced) Principals 

To explore heterogeneity in effectiveness among inexperienced principals, we estimated a 
separate post effect for each inexperienced principal who started at a school during the 2011-12, 
2012-13, 2013-14, or 2014-15 school years. Additional findings are in Appendix G. 

Figure 11 through Figure 14 show the distributions of principal effects in each district. The 
distributions are presented separately for principals trained by selected programs (treatment 
principals) and all other principals (comparison principals). Distributions are presented as box- 
and-whisker plots: the box represents the interquartile range of principal effects; the vertical line 
within the box represents the median principal effect; the whiskers represent the upper and lower 
adjacent values, where the whiskers are defined as plus or minus 1.5 times the interquartile range; 
and points beyond the upper and lower adjacent values represent outliers. In District A, we did 
not observe any variation in effectiveness among inexperienced principals. 

In most districts, the variation in effectiveness among inexperienced principals from selected 
programs was smaller than but of similar magnitude as the variation in effectiveness among all 
inexperienced principals (see Table 13). In reading/ELA, the smallest variation in inexperienced 
principal effectiveness occurred among principals from one of the selected programs in District A, 
where we did not observe any variation in effectiveness among inexperienced principals. We 
were unable to observe variation in principal effectiveness in District A because of the way our 
empirical Bayes estimates weighted the very small amount of information available from the 
small number of inexperienced principals we observed in District A, compared with the weight 
placed in the large variation in student test scores within each school following the arrival of the 
inexperienced principal. 11 


11 When we model the principal effects as fixed effects in District A, the individual effects have a range of 
about 0.45 SD in the normalized distribution of test scores. Although the fixed effects principal estimates 
are unbiased, we prefer the best linear unbiased predictors from the random effects, which minimize the 
mean squared error of the principal estimates. 
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Figure 11 . Distribution of Inexperienced Principal Effectiveness in Reading and 
Mathematics, by Treatment Status, District A 
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Figure 12 . Distribution of Inexperienced Principal Effectiveness in Reading, by Treatment 
Status, District B 
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Figure 13. Distribution of Inexperienced Principal Effectiveness in Reading, by Treatment 
Status, District C 
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Figure 14. Distribution of Inexperienced Principal Effectiveness in Reading, by Treatment 
Status, District D 
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Table 13. Variation in Inexperienced Principal Effectiveness by District, Subject, and 
Principal Preparation Program 


District 

Standard Deviation of Principal Effects: 
Reading/ELA 

Standard Deviation of Principal Effects: 
Mathematics 

Selected Program 

Other 

Selected Program 

Other 

A 

0.000 

0.000 

0.000 

0.000 

B 

0.062 

0.063 

0.013 

0.013 

C 

0.050 

0.082 

0.071 

0.100 

D 

0.157 

0.146 

0.186 

0.182 


The largest variation in inexperienced principal reading/ELA effectiveness occurred among 
treatment principals in District D, where a 1 SD increase in principal effectiveness was associated 
with a 0.157 SD increase in student achievement in reading. As an example, this finding suggests 
that replacing a below-average principal with an average principal would be associated with an 
average change across the school of about 6 test score points in reading achievement (assuming 
the SD of student test scores is about 38 points). The largest variation in inexperienced principal 
mathematics effectiveness occurred among treatment principals in District D, where a 1 SD 
increase in principal effectiveness was associated with a 0.186 SD increase in student achievement 
in mathematics. 

To put these estimates in perspective, a 1 SD increase in teacher effectiveness in reading is 
typically associated with a 0.11 SD increase in student achievement in reading, whereas a 1 SD 
increase in teacher effectiveness in mathematics is typically associated with a 0.15 SD increase in 
student achievement in mathematics (Hanushek & Rivkin, 2010). In Districts A, B, and C, our 
estimates of the impact of a 1 SD increase in principal effectiveness were typically about half as 
large as those reported by Hanushek and Rivkin (2010), whereas in District D our estimates were 
larger. 
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Effects of All Selected Program Principals on Student Achievement 

This section reports results of our analysis that includes all selected program graduates serving 
as principals in schools for at least 1 year from 2009-10 to 2014-15. 

We first provide demographic information for the students and schools included in the analysis. 
Next, we summarize the results across Districts A, B, C, and D, followed by more detailed results 
for all four districts. We were unable to obtain data to support this analysis in District E. 
Additional findings for Districts A, B, C, and D are in Appendix G. 

Students and Schools Included in the Analysis of All Principals 

District A. Table G-27 through Table G-33 in Appendix G summarize the characteristics of 
schools included in our analysis of the relative effectiveness of all principals from one of the 
selected programs in District A. Table 14 (also shown as Table G-28 in Appendix G) shows the 
years of experience for principals from one of the selected programs. For the year 2014-15, 
134 schools were included in the study, 11 of which were led by principals from one of the 
selected programs (Table G-28). The number of students included in the analysis averaged more 
than 55,000 per year (Table G-29). 

Table 14. Years of Experience Among Principals Who Were Trained by One of the Selected 
Programs in the Analysis of All Principals for District A 



More details about the achievement and demographic characteristics of students in schools led 
by principals from one of the selected programs are in Appendix G. 

District B. Table G-34 through Table G-39 in Appendix G summarize the characteristics of schools 
included in our analysis of the relative effectiveness of principals from one of the selected 
programs in District B. For the year 2014-15, 214 schools were included in the study, five of which 
were led by principals from one of the selected programs (Table G-34 and Table G-35). Table 15 
(also shown as Table G-34 in Appendix G) shows the years of experience for principals from one 
of the selected programs. The number of students included in the analysis averaged 
approximately 54,000 per year (Table G-36). 
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Table 15. Years of Experience Among Principals Who Were Trained by One of the Selected 
Programs in the Analysis of All Principals for District B 



Year 

Years of Experience as Principal — Treatment Principals 

Total Number of 

Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

0 

0 

0 

0 

0 

0 

0 

0 

2010-11 

0 

0 

0 

0 

0 

0 

0 

0 

2011-12 

0 

0 

0 

0 

0 

0 

0 

0 

2012-13 

2 

0 

0 

0 

0 

0 

0 

2 

2013-14 

2 

2 

0 

0 

0 

0 

0 

4 

2014-15 

2 

2 

1 

0 

0 

0 

0 

5 


District C. Table G-41 through Table G-47 in Appendix G summarize the characteristics of 
schools included in our analysis of the relative effectiveness of principals from one of the selected 
programs in District C. For the year 2014-15, 121 schools were included in the study, 26 of which 
were led by principals from one of the selected programs (Table G-41 and Table G-42). Table 16 
(also shown as Table G-42 in Appendix G) shows the years of experience for principals from one 
of the selected programs. The number of students included in the analysis averaged 
approximately 30,000 per year (Table G-43). 

Table 16. Years of Experience Among Principals Who Were Trained by One of the Selected 
Programs in the Analysis of All Principals for District C 




Years of Experience as Principal— Treatment Principals 

Total Number of 

Schools 



1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

5 

2 

2 

2 

0 

0 

0 

11 

2010-11 

3 

5 

2 

2 

2 

0 

0 

14 

2011-12 

3 

3 

3 

2 

2 

2 

0 

15 

2012-13 

5 

4 

5 

3 

2 

4 

0 

23 

2013-14 

4 

5 

4 

5 

2 

5 

0 

25 

2014-15 

4 

5 

5 

4 

5 

3 

0 

26 


District D. Table G-48 through Table G-54 in Appendix G summarize the characteristics of 
schools included in our analysis of the relative effectiveness of principals from one of the selected 
programs in District D. For the year 2013-14, 1,350 schools were included in the study, 183 of 
which were led by principals from one of the selected programs (Table G-48 and Table G-49). 
Table 17 (also shown as Table G-48 in Appendix G) shows the years of experience for principals 
from one of the selected programs. The number of students included in the analysis was greater 
than 440,000 per year (Table G-50). 
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Table 17. Years of Experience Among Principals Who Were Trained by One of the Selected 
Programs in the Analysis of All Principals for District D 


Year 

Years of Experience as Principal — Treatment Principals 

Total Number of 

Schools 

1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

27 

31 

21 

38 

35 

28 

0 

180 

2010-11 

22 

29 

32 

21 

31 

56 

0 

191 

2011-12 

13 

19 

33 

29 

18 

80 

0 

192 

2012-13 

17 

14 

19 

29 

23 

86 

0 

188 

2013-14 

14 

16 

13 

16 

25 

99 

0 

183 

2014-15 



Principals From Selected Programs: Impacts on Student Achievement 

When examining the impact of principals from selected programs (inexperienced and 
experienced) on student achievement, we looked at two variables: exposure and tenure. The 
exposure model allowed for the possibility, for example, that students with 1 year of attendance 
at a school led by a principal trained by one of the selected programs would be affected differently 
from students with 2 years of attendance at a school led by a principal trained by one of the 
selected programs. For the tenure model, we expected that the longer a principal's tenure at a 
school, the more time the principal has had to implement policies and other changes at the school. 

All Districts (A, B, C, D). Results for our models estimating the relative effectiveness of both 
experienced and inexperienced principals as it relates to both exposure and tenure can be 
summarized as follows (see Figure 15, Figure 16, Table 18, and Table 19; detailed by-district 
results can be found in Appendix G, Figure G-l to Figure G-4): 

• We found no evidence that principals in District A and District C from one of the selected 
programs were more or less effective, on average, at fostering reading and mathematics 
achievement than District A and District C principals from other programs. 

• We found no evidence that District B principals from one of the selected programs were 
more or less effective, on average, at fostering reading and mathematics achievement than 
District B principals from other programs. We found some evidence that District B 
principals may have been more effective at fostering reading achievement than principals 
from other programs. These results were based on only six principals from one of the 
selected programs in District B. 

• Evidence for District D suggested that principals from one of the selected programs may 
have been slightly less effective at fostering ELA achievement than District D principals 
from other programs. The effects of principals from one of the selected programs on 
mathematics achievement in District D were mixed. 
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Figure 15. Impact of Principals From Selected Programs on Student Reading/ELA 
Achievement, by Students' Years of Exposure and District 
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Figure 16. Impact of Principals From Selected Programs on Student Mathematical 
Achievement, by Students' Years of Exposure and District 



59 


Table 18. Impact of Principals From Selected Programs on Student Mathematical Achievement, by Students' Years of Exposure 
and District 


Covariate 

District A 

District B 

District C 

District D 

Reading 

Mathematics 

Reading 

Mathematics 

Reading 

Mathematics 

ELA 

Mathematics 

One or more years of exposure 

-0.010 

0.027 

0.030 

-0.099 

-0.021 

0.002 

-0.032** 

-0.019* 

(0.018) 

(0.031) 

(0.039) 

(0.088) 

(0.017) 

(0.019) 

(0.007) 

(0.009) 

Two or more years of exposure 

0.012 

0.009 

0.110* 

0.055 

-0.011 

-0.006 

0.002 

-0.001 

(0.013) 

(0.024) 

(0.055) 

(0.095) 

(0.012) 

(0.016) 

(0.006) 

(0.007) 

Three or more years of 
exposure 

0.016 

-0.036 



- 0.000 

-0.014 

-0.002 

-0.025** 

(0.018) 

(0.035) 



(0.018) 

(0.021) 

(0.006) 

(0.007) 

First year as principal 

-0.005 

-0.012 

-0.033** 

-0.074** 

-0.045** 

-0.069** 

-0.010* 

-0.036** 

(0.008) 

(0.013) 

(0.011) 

(0.015) 

(0.013) 

(0.015) 

(0.005) 

(0.007) 

Second year as principal 

-0.008 

-0.006 

-0.025* 

-0.044* 

-0.032* 

-0.056** 

-0.013* 

-0.036** 

(0.009) 

(0.014) 

(0.011) 

(0.017) 

(0.013) 

(0.016) 

(0.006) 

(0.007) 

Third year as principal 

-0.008 

-0.021 

-0.012 

-0.029 

-0.025 

-0.035* 

-0.010 

-0.028** 

(0.009) 

(0.015) 

(0.012) 

(0.019) 

(0.015) 

(0.017) 

(0.006) 

(0.008) 

Fourth year as principal 

0.003 

-0.018 

0.002 

-0.018 

-0.022 

-0.040* 

-0.009 

-0.022** 

(0.008) 

(0.013) 

(0.012) 

(0.017) 

(0.015) 

(0.018) 

(0.005) 

(0.007) 

Fifth year as principal 

0.002 

0.000 

-0.016 

-0.003 

-0.025 

-0.022 

0.001 

-0.001 

(0.007) 

(0.011) 

(0.012) 

(0.015) 

(0.015) 

(0.015) 

(0.005) 

(0.006) 

Experience missing 

0.003 

-0.001 

-0.046** 

-0.071** 

-0.017 

-0.119** 

0.019 

0.017 

(0.011) 

(0.020) 

(0.015) 

(0.024) 

(0.027) 

(0.028) 

(0.020) 

(0.020) 

Observations 

344,072 

345,545 

306,947 

372,607 

173,227 

177,487 

2,186,712 

2,215,866 

Pseudo-R-squared 

0.89 

0.90 

0.86 

0.84 

0.90 

0.92 

0.86 

0.86 


Robust standard errors, clustered within school-by-year cell, are in parentheses. 
**p< 0.01. ><0.05. 

See Table F-l in Appendix F for a complete list of model covariates. 
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Table 19. Impact of Principals From Selected Programs on Student Achievement, by Tenure at School 


Covariate 

District A 

District B 

District C 

District D 

Reading 

Mathematics 

Reading 

Mathematics 

Reading 

Mathematics 

ELA 

Mathematics 

First year of tenure 

-0.016 

-0.014 

0.011 

-0.008 

-0.002 

-0.019 

-0.022 

-0.017 

(0.008) 

(0.012) 

(0.016) 

(0.019) 

(0.014) 

(0.016) 

(0.015) 

(0.018) 

First year of tenure * treatment 
principal 

-0.004 

0.047 

-0.012 

-0.097 

-0.033 

0.004 

-0.015 

0.010 

(0.020) 

(0.038) 

(0.043) 

(0.101) 

(0.020) 

(0.025) 

(0.011) 

(0.015) 

Second year of tenure 

-0.009 

-0.001 

0.021* 

0.022 

-0.003 

-0.001 

-0.021 

-0.023 

(0.007) 

(0.013) 

(0.011) 

(0.019) 

(0.016) 

(0.016) 

(0.013) 

(0.015) 

Second year of tenure * treatment 
principal 

-0.006 

0.071 

0.029 

-0.020 

-0.033 

0.040 

-0.021 

0.026 

(0.022) 

(0.040) 

(0.058) 

(0.077) 

(0.025) 

(0.029) 

(0.013) 

(0.016) 

Three or more years of tenure * 
treatment principal 

-0.029 

0.027 

0.046 

-0.118 

-0.035 

0.022 

-0.024 

-0.015 

(0.020) 

(0.037) 

(0.077) 

(0.078) 

(0.026) 

(0.034) 

(0.013) 

(0.015) 

First year as principal 

0.007 

-0.001 

-0.041* 

-0.064** 

-0.050** 

-0.066** 

0.009 

-0.021 

(0.011) 

(0.017) 

(0.018) 

(0.021) 

(0.013) 

(0.015) 

(0.016) 

(0.019) 

Second year as principal 

-0.001 

-0.008 

-0.042** 

-0.062** 

-0.030 

-0.036 

0.002 

-0.018 

(0.011) 

(0.017) 

(0.014) 

(0.022) 

(0.016) 

(0.020) 

(0.014) 

(0.016) 

Third year as principal 

-0.006 

-0.022 

-0.011 

-0.028 

-0.023 

-0.036 

-0.009 

-0.025** 

(0.010) 

(0.015) 

(0.012) 

(0.019) 

(0.019) 

(0.021) 

(0.006) 

(0.008) 

Fourth year as principal 

0.005 

-0.018 

0.003 

-0.017 

-0.021 

-0.041* 

-0.008 

-0.020** 

(0.008) 

(0.013) 

(0.012) 

(0.016) 

(0.015) 

(0.018) 

(0.005) 

(0.007) 

Fifth year as principal 

0.005 

0.001 

-0.018 

-0.004 

-0.024 

-0.024 

0.002 

0.001 

(0.007) 

(0.011) 

(0.012) 

(0.015) 

(0.015) 

(0.015) 

(0.005) 

(0.006) 

Experience missing 

-0.003 

-0.006 

-0.042** 

-0.071** 

-0.018 

-0.119** 

-0.003 

0.002 

(0.012) 

(0.021) 

(0.015) 

(0.025) 

(0.027) 

(0.028) 

(0.026) 

(0.027) 

Observations 

344,072 

345,545 

306,947 

372,607 

173,227 

177,487 

2,186,712 

2,215,866 

Pseudo-R-squared 

0.89 

0.90 

0.86 

0.84 

0.90 

0.92 

0.86 

0.86 


Robust standard errors, clustered within school-by-year cell, are in parentheses. 
**p< 0.01. *p < 0.05. 

See Table F-l in Appendix F for a complete list of model covariates. 
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Discussion and Implications 

This study analyzed the relative effectiveness of graduates from five selected principal preparation 
programs who served as principals in their program's partner district between 2008-09 and 2014—15. 
These five programs were identified as having features thought to represent good preparation 
practice and are located throughout the United States. 

Although the principals reported generally positive perceptions of their preparation programs, 
we found little evidence that newly placed or more experienced principals from one of the 
selected programs were more or less effective, on average, than inexperienced principals trained 
by other programs. It is important to note that this finding does not indicate that these programs 
are not effective. It does mean that within the time period that we studied, we were unable to 
identify average effects of the programs on student achievement. 

We found significant variation in effectiveness among all newly placed principals. The principals 
also reported significant variation in the amount and types of supports provided by their districts 
(and varying levels of satisfaction with those supports). These results suggest that further 
exploration of the factors associated with variation in principal effectiveness may provide 
important insights on how best to develop excellent school principals. 

The potential for researchers and policymakers to identify factors associated with principal 
effectiveness may be limited, however, by the information available in state and local data 
systems. Another finding of this study was that high-quality information about principal 
preparation, education, experiences, and assignments was rarely available within districts (see 
also the related brief on this topic [George W. Bush Institute & American Institutes for Research, 
2016]). States and districts will need more and better information about principals and student 
outcomes, and programs themselves will need these data for their own improvement efforts. 

These results may point to several potential directions for further research, addressing questions 
such as the following: 

• What principal and preparation program characteristics are associated with greater 
principal effectiveness? 

• What types of supports after placement as a principal are associated with greater 
effectiveness? 

• How can we assess principal impact on other outcomes? 

• How does selection into programs vary, and what is its relationship to impact on student 
outcomes? 

• How does tenure in a school as a principal and in other school leadership positions relate 
to student outcomes? 
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Appendix A. Program Selection Criteria 

Introduction 

In developing principal preparation program selection criteria, AIR reviewed the research on best 
practices in principal preparation and synthesized the findings from several highly regarded 
institutions currently engaged in research and evaluation related to school leadership, including 
the following: 

• The Alliance to Reform Educational Leadership (AREL)'s nine principal preparation 
competencies 

• A study of exemplary leadership development programs commissioned by the Wallace 
Foundation and conducted by Linda Darling-Hammond at the Stanford Educational 
Leadership Institute in partnership with The Finance Project and WestEd 

• A report on case studies from these same exemplary programs commissioned by the 
Wallace Foundation and conducted by the Stanford Educational Leadership Institute in 
conjunction with the Finance Project 

• UCEA 's criteria for exemplary university-based educational leadership preparation 

• The Rainwater Leadership Alliance's principal preparation program competencies, 
described in their study on approaches to principal preparation 

AIR also considered study requirements, including program capacity to provide candidate data 
and intervention stability. 

Proposed Principal Preparation Program Selection Criteria 

1. Feasibility for Study 

• Program capacity to provide sufficient data impact (e.g., number of graduates per year 
and number of cohorts) 

• Program stability 

• Program includes certification process (i.e., participants do not enter the program having 
already completed a certification program through a university) 

• Program policy context, including state and district policies 

• Program participation in other studies 

2. Program Alignment to Research-Based Competencies. The program's structure and 
curriculum is organized according to a set of research-based standards and practices and 
displays alignment to these research-based competencies. 

3. Experiential Learning. Student learning (including coursework and other learning 
experiences) experiences expose students to the problems of school leadership practice, with 
the intent to build practical and technical knowledge. Learning experiences are scaffolded, 
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moving from classroom or online learning simulations to internship experiences where 
students lead all or a significant portion of a school's operations. 

4. High-Quality, Rigorous Recruitment and Selection. Vigorous recruitment of high-ability 
candidates with experience as expert, dynamic teachers and a commitment to instructional 
improvement. 

5. Early Years on-the-job Support. The program includes intensive on-the-job support (i.e., 
induction) for program graduates who are principals, such as mentoring, coaching, or a peer 
network for at least 1 year after graduation. 

6. Partnerships for Excellence. The program's administrative structure engages district 
personnel in (a) coteaching classes, (b) serving as sites for learning, (c) providing feedback 
on graduate quality, and (d) curriculum review and alignment. 

7. Evidence of Effects. The program can provide evidence of effects on advancing leadership 
practice, school culture, or student learning that have been developed through rigorous study. 
The program also has established routines for collecting program effectiveness information 
and uses data to adjust its programs. 
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Table A-l. Principal Preparation Criteria Comparison Matrix 


Proposed Selection 
Criteria 

AREL's Nine Principal 
Preparation Competencies 

Stanford University 
Educational Leadership 
Institute's Exemplary 
Leadership Program 
Criteria 

UCEA's Criteria for 
Exemplary University- 
Based Educational 
Leadership Preparation 
Award 

Rainwater Leadership 
Alliance's Principal 
Preparation Competency 
Framework 

1. Feasibility for Study 

• Program capacity to 
provide sufficient 
data impact (e.g., 
number of 
graduates per year 
and number of 
cohorts) 

• Program stability 

• Program policy 
context, including 
state and district 
policies 

• Program 
participation in 
other studies 

• Program includes 
certification process 

N/A 

N/A 

N/A 

N/A 

2. Program Alignment 
to Research-Based 
Competencies. The 

program's structure 
and curriculum is 
organized according 
to a set of research- 

based standards and 
practices and displays 

Competency Framework. 

All program components 
are organized around a 
competency framework 
based on research and 

evidence of effective 
practice that defines the set 
of skills, knowledge, and 
dispositions in both 

Research-based content 
that is aligned with 
professional standards 
and focused on 
instruction, 
organizational 
development, and 
change management; 

Reflects current research 

on the features, content, 
and experiences 
associated with effective 
leadership preparation. 

Knowledge and Skills: Is 

the program anchored to a 
set of nationally 

Undergirding competency 
framework: A competency 
framework embodies the 
set of skills, knowledge, 
and dispositions that an 
effective principal needs in 
order to drive student 

achievement. The 
competency framework 
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Proposed Selection 
Criteria 

AREL's Nine Principal 
Preparation Competencies 

Stanford University 
Educational Leadership 
Institute's Exemplary 
Leadership Program 
Criteria 

UCEA's Criteria for 
Exemplary University- 
Based Educational 
Leadership Preparation 
Award 

Rainwater Leadership 
Alliance's Principal 
Preparation Competency 
Framework 

alignment to these 

research-based 

competencies. 

instructional and 
operational domains that a 
principal must have to 
drive high levels of student 
achievement. 

Curricular coherence 
that links goals, learning 
activities, and 
assessments around a set 
of shared values, beliefs, 
and knowledge about 
effective organizational 
practice; 

recognized leadership 
standards? How does the 
program integrate research 
and professional 
knowledge with leadership 
practice? In what ways are 
issues related to leading 
diverse and/or low-income 
student populations dealt 
with? 

serves as the standards for 
the program and all 
program components — 
recruiting, selection, 
training, support, and 
evaluation— are aligned 
and designed to ensure 
that graduates leave with 
these requisite 
competencies. 

3. Experiential 

Learning: Student 
learning (including 
coursework and other 
learning experiences) 
experiences expose 
students to the 
problems of school 
leadership practice, 
with the intent to 
build practical and 
technical knowledge. 
Learning experiences 
are scaffolded, 
moving from 
classroom or online 
learning simulations 
to internship 
experiences where 

Coursework. Program 
coursework is aligned to 
the competency framework 
and is highly experiential 
and applied, with 
candidates receiving 
frequent feedback and 
assessment about their 
performance and progress 
toward acquisition of the 
competencies. 

Clinical Leadership 
Experiences. The program 
provides candidates 
leadership opportunities 
with real responsibilities for 
driving student achievement 
gains, leading adults, and 

Field-based internships 
that enable candidates to 
apply leadership 
knowledge and skills 
under the guidance of an 
expert practitioner; 

Problem-based learning 
strategies, such as case 
methods, action research, 
and projects, that link 
theory and practice and 
support reflection; 

Reflects current research 

on the features, content, 
and experiences 
associated with effective 
leadership preparation. 

Internship: How does the 
internship support 
leadership development? 
How is it supervised and 
by whom? What does it 
involve and how is it 

funded? 

Relevant and practical 
coursework: Course 
content is aligned to the 
competency framework 
with a strong emphasis on 
instructional leadership, 
human capital performance 
management, and school 
culture. The delivery is 
practical and applied— not 
merely theoretical— and 
allows aspiring principals to 
practice their skills and 
approximate real-life, on- 
the-job situations through 
role-play, case studies, and 
simulations. 
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Proposed Selection 
Criteria 

AREL's Nine Principal 
Preparation Competencies 

Stanford University 
Educational Leadership 
Institute's Exemplary 
Leadership Program 
Criteria 

UCEA's Criteria for 
Exemplary University- 
Based Educational 
Leadership Preparation 
Award 

Rainwater Leadership 
Alliance's Principal 
Preparation Competency 
Framework 

students lead all or a 
significant portion of a 
school's operations. 

performing an array of 
leadership functions with 
structured coaching or other 
mechanisms to support their 
reflection, practice and 
growth. 



Experiential, clinical 
school-based 
opportunities: Through 
partnerships with school 
districts or charter schools 

for clinical school-based 
experiences, trainees are 
given authentic 
opportunities to test their 
leadership mettle in school 
settings over a significant 
period of time (at least six 
Principal preparation 
programs and standards 
falling short 1 

www.americanprogress.org 
11 months), while receiving 
support and feedback from 
experienced mentors and/or 
coaches. Programs expect 
their trainees to 
demonstrate proficiency in 
the competency framework 
areas as a requirement for 
graduation. 
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Proposed Selection 
Criteria 

AREL's Nine Principal 
Preparation Competencies 

Stanford University 
Educational Leadership 
Institute's Exemplary 
Leadership Program 
Criteria 

UCEA's Criteria for 
Exemplary University- 
Based Educational 
Leadership Preparation 
Award 

Rainwater Leadership 
Alliance's Principal 
Preparation Competency 
Framework 

4. High-Quality, 

Rigorous Recruitment 
and Selection. 

Vigorous recruitment 
of high-ability 
candidates with 
experience as expert, 
dynamic teachers and 
a commitment to 

instructional 

improvement. 

Recruitment. The program 
has a proactive, targeted 
recruitment strategy that 
attracts an expanded 
candidate pool with skills 
and dispositions aligned 
with the competency 
framework. 

Candidate Selection. The 

program uses a rigorous 
selection process aligned to 
the competency framework 
that only admits those 
candidates who 

demonstrate the 
knowledge, skills, and 
dispositions required to 
develop into effective 
school leaders. 

Vigorous recruitment of 
high-ability candidates 

with experience as 
expert, dynamic teachers 
and a commitment to 

instructional 

improvement; 

Recruitment: What 
strategies are used to 
recruit candidates? What 
perspectives, priorities 
and data inform the 
development of 
recruitment materials? 

Who participates in the 
recruitment process and 
why? 

Selection: What 
strategies, information 
and criteria are used to 

select candidates for 
participation in the 
preparation program? 

How is the selection 
criteria and process 
integral to the program's 
goals and approach? Who 
participates in candidate 
selection and how? Does 
the student body reflect 
the diversity of the area 
served by the program? 

Strategic and proactive 
recruiting: Strategic and 
proactive recruitment of 
high-potential candidates 
is critical. High-quality 
candidates with the skills 
and dispositions aligned to 
the competency 
framework are identified 
and targeted, which in 
turn provides programs 
strong candidate pools. 

Rigorous selection 
process: Candidates are 
required to demonstrate 
their skills, knowledge, 
and dispositions through a 
rigorous, multistep 
selection process. A series 
of real-time performance- 
based assessments is used 

to amass data on 

candidates in order to 
spotlight those with the 
highest potential to be best 
prepared for success as 
principals. 
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Proposed Selection 
Criteria 

AREL's Nine Principal 
Preparation Competencies 

Stanford University 
Educational Leadership 
Institute's Exemplary 
Leadership Program 
Criteria 

UCEA's Criteria for 
Exemplary University- 
Based Educational 
Leadership Preparation 
Award 

Rainwater Leadership 
Alliance's Principal 
Preparation Competency 
Framework 

5. Early Years on-the- 
Job Support. The 

program includes 
intensive on-the-job 
support (i.e., 
induction) for 
program graduates 
who are principals, 
such as mentoring, 
coaching, or a peer 
network for at least 

1 year after 
graduation. 

Post-Graduate Support. 

The program provides 
tailored support for placing 
graduates and ensuring 
their success during at least 
their first year as a school 
principal. 

Mentoring or coaching 
that supports modeling, 
questioning, 
observations of practice, 
and feedback; 

Reflects current research 

on the features, content, 
and experiences 
associated with effective 
leadership preparation. 

Supportive Structures. 

What program structures 
(e.g., cohorts, mentoring, 
coaching) are provided to 
support communities of 
practice? 

Placement and on-the-job 
support: Upon successful 
completion of the 
program, trainees are 
given further assistance on 
their road to effective 
school leadership. 

Programs assist their 
graduates in identifying 
and securing school 
leadership positions and 
provide continued support 
to help them grow and be 
effective on the job. 

6. Partnerships for 
Excellence. The 

program's 
administrative 
structure engages 
school district 
personnel in (a) co- 
teaching classes, 

(b) serving as sites for 
learning, (c) providing 
feedback on graduate 
quality, and 
(d) curriculum review 
and alignment. 

Context. The program 
partners with local school 
districts or other operators 
of schools to influence the 

conditions and level of 
decision-making authority 
necessary for the principal 
to impact student 
achievement successfully. 
Throughout this document, 
the term "districts" should 
be interpreted to include all 
organizations that operate 
schools, including charter 
schools and private schools. 

Collaboration between 

universities and school 

districts to create 

coherence between 
training and practice as 
well as pipelines for 
recruitment, preparation, 
hiring, and induction. 

Partnerships: What kinds 
of partnerships inform the 
program? How have 
district personnel 
influenced and/or 
informed the program? 
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Proposed Selection 
Criteria 

AREL's Nine Principal 
Preparation Competencies 

Stanford University 
Educational Leadership 
Institute's Exemplary 
Leadership Program 
Criteria 

UCEA's Criteria for 
Exemplary University- 
Based Educational 
Leadership Preparation 
Award 

Rainwater Leadership 
Alliance's Principal 
Preparation Competency 
Framework 

7. Evidence of Effects. 

The program can 
provide evidence of 
effects on advancing 
leadership practice, 
school culture, or 
student learning that 
have been developed 
through rigorous 
study. The program 
also has established 
routines for collecting 
program effectiveness 
information and uses 
data to adjust 
programs. 

Evaluation. The program 
actively collects data to 
continuously improve its 
own performance and 
measure the effectiveness of 
its graduates. 


Has demonstrated 

evidence of program 
effectiveness. 

Program Improvement: 

How are candidate and 
program assessments 
used to promote program 
improvement? 

Robust data collection 
and continuous learning: 

To continuously improve 
the program and ensure 
trainee effectiveness, data 
are collected and 
monitored during the 
program and after 
graduation. Programs 
track the effectiveness of 
their principal graduates 
on student achievement 
and school performance 
over time. This vision 
differs dramatically from 
the traditional university- 
based master's in 
educational leadership 
program that prepares 
most of our country's 
principals for the job. 

There is little match 

between what we now 
know of effective program 
elements and the elements 
of many university 
programs that are 
approved by states. 
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Source Information: Complete List of Standards From Each Organization 

AREL's Nine Principal Preparation Competencies (From 
http://www.bushcenter.org/sites/default/files/ninecompetencies.pdf) 

1. Program Purpose The program is designed for the express purpose of producing and 
placing school principals who dramatically improve student learning and sustain that 
improvement. While graduates of school leadership programs may also serve in other 
leadership capacities at the campus and district level, the focus of the AREL project is on 
preparing and empowering principals. 

2. Competency Framework All program components are organized around a competency 
framework based on research and evidence of effective practice that defines the set of skills, 
knowledge, and dispositions in both instructional and operational domains that a principal 
must have to drive high levels of student achievement. 

3. Recruitment The program has a proactive, targeted recruitment strategy that attracts an 
expanded candidate pool with skills and dispositions aligned with the competency 
framework. 

4. Candidate Selection The program uses a rigorous selection process aligned to the 
competency framework that only admits those candidates who demonstrate the knowledge, 
skills and dispositions required to develop into effective school leaders. 

5. Coursework Program coursework is aligned to the competency framework and is highly 
experiential and applied, with candidates receiving frequent feedback and assessment about 
their performance and progress toward acquisition of the competencies. 

6. Clinical Leadership Experiences The program provides candidates leadership 
opportunities with real responsibilities for driving student achievement gains, leading 
adults, and performing an array of leadership functions with structured coaching or other 
mechanisms to support their reflection, practice and growth. 

7. Post-Graduate Support The program provides tailored support for placing graduates and 
ensuring their success during at least their first year as a school principal. 

8. Context The program partners with local school districts or other operators of schools to 
influence the conditions and level of decision-making authority necessary for the principal 
to impact student achievement successfully. Throughout this document, the term "districts" 
should be interpreted to include all organizations that operate schools, including charter 
schools and private schools. 

9. Evaluation The program actively collects data to continuously improve its own 
performance and measure the effectiveness of its graduates. 


Exemplary Leadership Program Criteria From the Stanford Educational Leadership 
Institute (From http://www.wallacefoundation.org/knowledge-center/school- 
leadership/key-research/Documents/Preparing-School-Leaders.pdf [p. 68]) 

1. Research-based content that is aligned with professional standards and focused on 
instruction, organizational development, and change management; 

2. Curricular coherence that links goals, learning activities, and assessments around a set of 
shared values, beliefs, and knowledge about effective organizational practice; 

3. Field-based internships that enable candidates to apply leadership knowledge and skills 
under the guidance of an expert practitioner; 

4. Problem-based learning strategies, such as case methods, action research, and projects, that 
link theory and practice and support reflection; 

5. Cohort structures that enable collaboration, teamwork, and mutual support; 

6. Mentoring or coaching that supports modeling, questioning, observations of practice, and 
feedback; 

7. Collaboration between universities and school districts to create coherence between training 
and practice as well as pipelines for recruitment, preparation, hiring, and induction (Davis, 
Darling-Hammond, LaPointe & Meyerson, 2005; Jackson & Kelley, 2002). 

Several other factors appeared to contribute to program effectiveness, including: 

• Vigorous recruitment of high-ability candidates with experience as expert, dynamic 
teachers and a commitment to instructional improvement; 

• Financial support for pre-service candidates to enable them to undertake an intensive 
program with a full-time internship; and 

• District or state infrastructure that supports specific program elements and embeds 
programs within a focused school reform agenda. 

UCEA's Criteria for Exemplary University-Based Educational Leadership Preparation 
Award (From http://www.ucea.org/opportunities/exemplary-educational-leadership- 
preparation/) 

The recipient of this award is generally selected based on the extent to which the program: 

1. Reflects current research on the features, content, and experiences associated with effective 
leadership preparation, and 

2. Has demonstrated evidence of program effectiveness. 
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Selection criteria for this award specifically include detailed information related to the following 
questions: 

• Program Focus: What kind of leader does the program claim to prepare and how is that 
type of leadership reflected in the various dimensions of the program (e.g., recruitment, 
curriculum, practical experiences, and assessment)? 

• Recruitment: What strategies are used to recruit candidates? What perspectives, priorities 
and data inform the development of recruitment materials? Who participates in the 
recruitment process and why? 

• Selection: What strategies, information and criteria are used to select candidates for 
participation in the preparation program? How is the selection criteria and process 
integral to the program's goals and approach? Who participates in candidate selection and 
how? Does the student body reflect the diversity of the area served by the program? 

• Learning Experiences: What is the point of view about learning in the program? How 
does teaching reflect this perspective? What kinds of learning experiences are integrated 
into the program? 

• Knowledge and Skills: Is the program anchored to a set of nationally recognized 
leadership standards? How does the program integrate research and professional 
knowledge with leadership practice? In what ways are issues related to leading diverse 
and/or low-income student populations dealt with? 

• Internship: How does the internship support leadership development? How is it 
supervised and by whom? What does it involve and how is it funded? 

• Supportive Structures: What program structures (e.g., cohorts, mentoring, coaching) are 
provided to support communities of practice? 

• Partnerships: What kinds of partnerships inform the program? How have district 
personnel influenced and/or informed the program? 

• Candidate Assessment: How do you know that candidates' are gaining the intended 
knowledge, skills and dispositions? How are candidate assessments used by to support 
candidate growth? 

• Program Improvement: How are candidate and program assessments used to promote 
program improvement? 

• Faculty: How many faculty teach in this program? Do program faculty represent expertise 
from the research and practice communities? How does the program ensure that its 
faculty have the capabilities to prepare effective educational leaders? How do faculty 
members work together to design, improve and deliver the program? 
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The Rainwater Leadership Alliance's Key Principal Preparation Program Design 

Elements (From http://www.anewapproach.org/download.html) 

1. Undergirding competency framework: A competency framework embodies the set of skills, 
knowledge, and dispositions that an effective principal needs in order to drive student 
achievement. The competency framework serves as the standards for the program and all 
program components— recruiting, selection, training, support, and evaluation— are aligned 
and designed to ensure that graduates leave with these requisite competencies. 

2. Strategic and proactive recruiting: Strategic and proactive recruitment of high-potential 
candidates is critical. High-quality candidates with the skills and dispositions aligned to the 
competency framework are identified and targeted, which in turn provides programs strong 
candidate pools. 

3. Rigorous selection process: Candidates are required to demonstrate their skills, knowledge, 
and dispositions through a rigorous, multistep selection process. A series of real-time 
performance-based assessments is used to amass data on candidates in order to spotlight 
those with the highest potential to be best prepared for success as principals. 

4. Relevant and practical coursework: Course content is aligned to the competency 
framework with a strong emphasis on instructional leadership, human capital performance 
management, and school culture. The delivery is practical and applied— not merely 
theoretical— and allows aspiring principals to practice their skills and approximate real-life, 
on-the-job situations through role-play, case studies, and simulations. 

5. Experiential, clinical school-based opportunities: Through partnerships with school 
districts or charter schools for clinical school-based experiences, trainees are given authentic 
opportunities to test their leadership mettle in school settings over a significant period of 
time (at least six Principal preparation programs and standards falling short I 
www.americanprogress.org 11 months), while receiving support and feedback from 
experienced mentors and/or coaches. Programs expect their trainees to demonstrate 
proficiency in the competency framework areas as a requirement for graduation. 

6. Placement and on-the-job support: Upon successful completion of the program, trainees 
are given further assistance on their road to effective school leadership. Programs assist 
their graduates in identifying and securing school leadership positions and provide 
continued support to help them grow and be effective on the job. 

7. Robust data collection and continuous learning: To continuously improve the program 
and ensure trainee effectiveness, data are collected and monitored during the program and 
after graduation. Programs track the effectiveness of their principal graduates on student 
achievement and school performance over time. This vision differs dramatically from the 
traditional university-based master's in educational leadership program that prepares most 
of our country's principals for the job. There is little match between what we now know of 
effective program elements and the elements of many university programs that are 
approved by states. 
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Appendix B. Program Documentation Request: Evaluating the 
Impact of Principal Preparation Program Documentation Request 

Overview 

As described in more detail in the enclosed brochure, American Institutes for Research (AIR) is 
conducting a study on innovative principal preparation programs serving a small number of 
districts across the United States. The initial stage of AIR's analysis will describe the 
characteristics of schools within identified districts led by principals from selected principal 
preparation programs, along with the characteristics of the programs themselves. The second 
stage of the analysis will estimate the effects that principals from the selected programs have had 
on student and school outcomes. This letter outlines AIR's request for information from your 
program for this important study. 

Table B-l describes the types of information we are seeking. For each type, we ask that you submit 
relevant documentation that provides more detail than that which is publicly available on your 
program's website or which may not be publicly available at all. In the column to the right, please 
note which documents correspond with each type and send this table along with your 
submission. 

Document Format and Submission 

All documents and files must be clearly labeled with your institution's name, the name and 
description of the document, and the date (e.g., PROGRAM NAME_2014Syllabus_06.12.14). This 
will help us easily store and identify your documents. You can e-mail documents in a compressed 
Zip file to Dana Chambers at dchambers@air.org . Please send all available documents by 
Monday, July 28, 2014. 

For More Information 

To find out more or discuss questions or concerns, please contact Dana Chambers at 
dchambers@air.org or 202-403-6899. 


Table B-l. Documentation Request 


Type of Information 
Requested 

Example Description or Content 

Document and 

File Name 

Description of 
Principal 

Preparation Program 
and Components 

■ Program description and structure 

■ Program mission and goals 

■ Degrees or certifications offered 

■ Duration of the program 

■ Typical time between graduation and placement as a 
principal 
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Type of Information 
Requested 

Example Description or Content 

Document and 

File Name 

Graduation 

Information and 

Statistics 

■ Graduation requirements, including capstone and 
internship requirements 

■ Graduation statistics (e.g., number of previous 
graduates from the program for the past 6 years, 2009- 
2014) 

■ Percentage or number of graduates who are placed as 
principals in partner district 


Program Alignment 
With Competencies 

■ Description of standards or competencies with which 
the program is aligned 

■ Documentation of alignment of curriculum or 
coursework with program competencies or standards 

■ Documentation of alignment of field-based learning 
experiences with program competencies or standards 


Experiential 

Learning 

■ Course description or syllabi 

■ Student course sequence 

■ Description of any experiential learning components of 
coursework 

■ Description of any field-based learning experiences 
(e.g., internships, observations, or residencies) 


Faculty Information 

■ Curricula vitae of faculty, including professional status 
and courses taught 


Recruitment and 

Selection 

Information 

■ Program recruitment materials for candidates 

■ Documentation regarding program's recruitment 
process or any recruitment initiatives 

■ Minimum program acceptance requirements (e.g., 
assessment scores, grade point average, or years of 
teaching experience) 

■ Documentation of leadership competencies or qualities 
sought in candidates 

■ Information regarding application process 

■ Information regarding interview and hiring process 


Early Years on-the- 
Job Support 

■ Description of university's mentorship and coaching 
support offerings for program graduates 

■ Description of district's mentorship and coaching 
support offerings for program graduates 

■ Timelines for coaching and mentoring 

■ Goals, expectations, and requirements for coaching or 
mentoring component of program 

■ Qualifications and experience (e.g., curricula vitae) of 
mentors and coaches 
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Type of Information 
Requested 

Example Description or Content 

Document and 

File Name 

Partnership 

Information 

■ Documents describing partnerships with local school 
districts 

■ Documentation regarding any district personnel 
involvement in program (as mentors, course 
instructors, or other involvement) 

■ Documentation regarding any district personnel 
involvement in curriculum development, review, or 
refinement 

■ Number of existing district partners 

■ Years of current partnerships 


Evaluation and 

Research 

■ Description of any participation in research or 
evaluation with other partner organizations 

■ Description of any evaluation or assessment of 
program or graduate effectiveness 

■ Reports data or outcomes of any research or evaluation 
of program or graduates 
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Appendix C. Program Administrator Interview Protocol: 
Evaluating the Impact of Principal Preparation Program Staff 

Interview Protocol 

Introduction 

Hello, I'm from American Institutes for Research, which conducts educational 

research and evaluations. I am a member of a team that is working with the George W. Bush 
Institute to study the impact of principal preparation programs and the districts they partner with 
on graduates' leadership abilities, the schools that they lead, and the students that they serve. The 
purpose of this interview is to understand more about how your program prepares school 
principals. We have already reviewed publicly available information about your program (and 
documentation provided by the program, if applicable); this interview is intended to be a follow- 
up to help us gather additional details. 

If at any point during this interview you feel that you do not have the information required to 
answer the question, or you feel that someone else in the program may be better able to answer 
the question, please let me know and we will make a note to follow up with that person directly 
for more information. 

Confidentiality 

This interview should take 45-60 minutes. Your responses will be used to add detail to the 
description of your program's characteristics and provide context for the quantitative analysis of 
impact. We will not identify you by name in our reporting of findings, although it may be possible 
for some individuals to infer your identity. 

You do not have to participate in this interview if you do not want to, but we hope you will 
because we value your perspective. If you decide to participate, you may withdraw at any time 
without penalty or risk of harm to your relationship with the district. You may refuse to answer 
any question. 

I will be taking notes as we talk and would like to record our conversation to ensure accuracy. 
May I have your permission to record this conversation? 

Respondent's permission given for audio recording: Q Yes Q No 

NOTE TO INTERVIEWER: IF ANSWER IS YES, TURN ON VOICE RECORDER AND 
PROCEED. Repeat your request for permission to record after turning on the audio recorder so 
permission is documented. For purposes of labeling the recording, note the respondent's position 
and program. 
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General Program Background and Respondent Information 

1. What is your current position? 

a. Probe: How long have you worked in your current position ? 

b. Probe: How long have you been at this program? 

2. Is it okay to refer to your program as [INSERT NAME] in our description? 

Feasibility and Context 

3. How long has this program been operating? 

a. Probe: Given that programs can change, can you tell me how long the program has been in 
place in its current form? 

b. Probe: Do you anticipate that a redesign of the program will occur during the next 2 years? 

4. How many graduates has your program had in the past 6 years? 

a. Probe: What is the average cohort size (or if not organized by cohort, principal preparation 
program matriculation numbers) for the program during the past 6 years (2009-2014)? 

b. Probe: How many candidates do you anticipate enrolling to start your program in fall 2014? 

c. Probe: What percentage of candidates enrolled in the program would be considered full-time 
students and what percentage would be considered part-time students? 

d. Probe: What percentage of candidates does the program consider to be on track for timely 
graduation? 

5. Is your program currently participating in any other research studies or evaluations? 

a. Probe: Can you describe the nature of any other studies or evaluations in which you are or 
will be participating? 

6. Are any other school leadership initiatives taking place in the district in which your 
program places graduates as school leaders? If yes, please describe. 

7. What is the percentage of program graduates who obtain principal positions in 
elementary or secondary schools immediately after graduation? 

a. Probe: What is the percentage of program graduates that obtain principal positions in 
elementary or secondary schools within 3 years of graduation? 

b. Probe: How do you know the data are accurate? 
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Alignment With Principal Preparation Selection Criteria 


Program Alignment to Professional Standards or Competencies 

8. To what degree is the program aligned with professional standards or competencies? 

a. Probe: Describe the standards or competencies that underlie your program. 

b. Probe: Why did you choose these standards or competencies? 

c. Probe: How do you know the standards or competencies are integrated into coursework? 

d. Probe: How are other elements of the program— such as field-based learning experiences or 
post-graduate support— aligned with these competencies? 

e. Probe: How do you know if your graduates achieved these competencies? 

f. Probe: Has your program made any significant changes to its competencies during the past 6 
years? If yes, please describe. 

9. How is the coursework in your program structured? 12 

a. Probe: What is the order and sequencing of the coursework in your program, and on what 
basis was this structure determined? 

b. Probe: How, if at all, is the coursework integrated with field-based learning experiences? 

c. Probe: What specific instructional strategies does your program use repeatedly in its 
coursework? 

Experiential Learning 

10. To what degree does the program structure incorporate experiential or field-based learning? 

a. Probe: Wl\at approaches does the program use? Why does the program use these approaches? 

b. Probe: Does the coursework include simulations of leadership experience? 

c. Probe: Does the program include field-based opportunities such as internships, school 
observations, fieldwork, or other opportunities for students to apply their learning in real- 
world settings? If so, please describe the amount of time that candidates spend in internship 
schools and the tasks that they complete in those schools. In addition, tell us about the 
qualifications of coaches and mentors who support candidate internship experiences. 

d. Probe: Given that you have an internship experience, can you tell me how candidates' 
experiences vary during the internship? For example, candidates may do the same activities 
throughout the experience or they may have escalating leadership responsibilities culminating 
in leading the entire school for a period of time. 

e. Probe: Has your program made any significant changes to its experiential learning 
component during the past 6 years? If yes, please describe. 

12 See also Darling-Hammond et al. (2007), p. 177. 
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Recruitment and Selection 

11. Which strategies, if any, do you use to recruit candidates for the program? 

a. Probe: Does your program actively recruit candidates from outside the school district in 
which you place principals? 

b. Probe: Are most applicants self-selected or do other recruitment initiatives exist? 

c. Probe: What are candidates' typical characteristics? (e.g., what type of experience or role 
responsibilities do they have)? 

d. Probe: Has your program made any significant changes to its recruitment process during the 
past 6 years? If yes, please describe. 

12. What is the selection process for applicants? 

a. Probe: What are the minimum admission requirements? 

b. Probe: Do applicants participate in any scenarios or other interactive simulations during the 
interview process? 

c. Probe: Are applicants screened for any particular qualities associated with effective school 
leadership beyond academic and experience-based criteria (e.g., transcripts, years of effective 
teaching experience, or letters of recommendation)? In what ways? 

d. Probe: Has your program made any significant changes to its selection process during the 
past 6 years? If yes, please describe. 

Early Years on-the-job Support 

13. What types of postgraduate or induction support does the program provide its 

graduates? 

a. Probe: Are graduates provided with coaches or mentoring? If so, for how long? 

b. Probe: What are the qualifications and experience of any mentors or coaches that support 
your graduates? 

c. Probe: Does your program provide any peer networking opportunities for new principals 
after graduation ? If so, please describe. 

d. Probe: Does your program provide any professional development or training opportunities to 
new principals after graduation? If so, please describe. 

c. Probe: Has your program made any significant changes to its induction support during the 
past 6 years? If yes, please describe. 
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Partnerships for Excellence 

14. Does the program engage in any partnerships with districts or other organizations to 

support program effectiveness? 

a. Probe: Do district personnel teach any of the classes in the program? 

b. Probe: Do districts provide sites or schools for field-based learning? 

c. Probe: Do districts or other organizations provide feedback on or evaluation of your graduates 
or program participants? 

d. Probe: Are districts or other organizations involved in the development or alignment of your 
program curriculum? 

e. Probe: Have any significant changes been made to the district's partnership with your 
program during the past 6 years? If yes, please describe. 

Evaluation of Effectiveness 

15. How does your program evaluate its effectiveness? 13 

a. Probe: What data does your program collect to measure program quality and effectiveness? 
How are these data analyzed? 

b. Probe: How frequently does your program collect and analyze effectiveness data? 

c. Probe: Are your evaluations conducted internally or externally? By whom? 

d. Probe: To what extent, if at all, does your program use evaluation data to make changes to the 
program? 

e. Probe: Has your program made any changes to the way it evaluates program effectiveness 
during the past 6 years? If yes, please describe. 

16. How does your program facilitate continuous improvement? 14 

a. Probe: Does your program engage in course syllabi review and revision ? If so, how 
frequently? 

b. Probe: What processes does your program have in place to review and revise course syllabi for 
alignment with program standards and competencies (i.e., who reviews, what is the review 
process, and what is the revision process)? 

17. Do you have any questions or comments you would like to add? 

Thank you for a rich discussion today. We really appreciate the feedback you have given us. 


13 See also Darling-Hammond et al. (2007), p. 186. 

14 See also Darling-Hammond et al. (2007), p. 179. 
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Appendix D. Principal Interview Protocol: Evaluating the Impact 
of Principal Preparation: Principal Interview Protocol 

Introduction 

Hello, I'm from American Institutes for Research, which conducts educational 

research and evaluations. I am a member of a team that is working with The George W. Bush 
Institute to study the impact of principal preparation programs and the districts they partner with 
on graduates' leadership abilities, the schools they lead, and the students they serve. We have 
already collected publicly available information about the principal preparation program you 
attended and the district where you serve as a principal, and we have conducted interviews with 
program and district personnel. The goals of this principal interview are as follows: 

• To verify any information we received from the principal preparation program you 
attended and the district where you currently work, including your general background 
and experience and your tenure at your current school. 

• To determine if your school or district is participating in any special initiatives that would 
affect the outcomes of our research. 

• To understand other supports you may be receiving from the district, the program you 
attended, or other sources that may influence your leadership of the school. 

• To gather data that will help us understand the ways in which you have applied the 
learning acquired during your training program or induction as a new principal. 

If at any point during this interview you feel that you do not have the information required to 
answer the question, please let me know. 

Confidentiality 

This interview should take 30-45 minutes. Your responses will be used to add detail to the 
description of your program's training and district's principal placement and support processes, 
and they will provide context for the quantitative analysis of impact. We will not identify you by 
name in our reporting of findings, but it is possible that some individuals may infer your identity. 

You do not have to participate in this interview if you do not want to, but we hope you will 
because we value your perspective. If you do decide to participate, you may withdraw at any 
time without penalty or risk to your relationship with the district or program. You may refuse to 
answer any question. 

I will be taking notes as we talk and would like to record our conversation to ensure accuracy. 
May I have your permission to record this conversation? 

Respondent's permission given for audio recording: Q Yes Q No 
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NOTE TO INTERVIEWER: IF ANSWER IS YES, TURN ON VOICE RECORDER AND 
PROCEED. Repeat your request for permission to record after turning on the audio recorder so 
permission is documented. For the purpose of labeling the recording, note the respondent's 
position and program. 

General Principal Background and Data Confirmation 

1. Where did you receive your principal preparation training? 

a. Probe: In what year were you admitted to [PROGRAM NAME]? 

b. Probe: When did you graduate from [PROGRAM NAME]? 

c. Probe: From the time of graduation, about how long did it take for you to be hired and placed 
as a principal? 

2. What is your current position? 

a. Probe: How long have you worked in your current position? (Note: Confirm start date.) 

b. Probe: How long have you been a principal ? 

c. Probe: What other school-based roles have you served in throughout your career? (Note: 
Clarify any other school leadership experience, including assistant principalships or previous 
principal experience.) 

Alignment With Principal Preparation Selection Criteria 

Alignment to Professional Standards or Competencies 

Note to Interviewer: Please preface the next two questions and all Likert scale questions with the 
following: "For the next question. I'm going to ask you to give a rating, and there are three 
options: not at all, to a moderate extent, or to a great extent." 

3. To what extent did you feel that [PROGRAM NAME] was aligned to a clear set of 
competencies or standards related to the qualities of an effective principal? Would you 
say the program was not aligned to standards at all, was aligned to a moderate extent, or 
was aligned to a great extent? Please explain your answer. 

a. Probe: Describe the standards or competencies that are used by the program you attended. 

4. To what extent did the content of your course work support your practice as a principal? 
Would you say the coursework did not support your practice as a principal at all, 
supported your coursework to a moderate extent, or supported your practice to a great 
extent? Please provide specific examples. 
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Experiential Learning 

5. Did your principal preparation program include any experiential learning opportunities 
(such as preservice residencies, internships, or observations)? 

6. Describe your overall experience with any experiential learning opportunities during the 
program. 

a. Probe: To what extent, if at all, do you feel that the preservice experiential learning 
opportunities prepared you to be an effective principal? Would you say the preservice 
learning opportunities did not prepare you at all, prepared you to a moderate extent, or 
prepared you to a great extent? Please explain your answer. 

b. Probe: To what extent, if at all, do you feel that the preservice experiential learning 
opportunities facilitated your placement as a principal in the district? Would you say the 
preservice learning opportunities did not facilitate your placement at all, facilitated your 
placement to a moderate extent, or facilitated your placement to a great extent? Please 
explain your answer. 

Program Recruitment and Selection 

7. How did you learn about [PROGRAM NAME]? 

8. Please briefly tell me about the selection process for [PROGRAM NAME]. 

District Recruitment, Selection, and Placement 

9. Tell me how you ended up in your current principal position, including the recruitment 
and hiring processes. 

Early Years on-the-job Support 

10. What types of support did you receive from as a new principal, and how long did you 
receive this support? 

a. What types of support did you receive from your preparation program? 

b. What types of support did you receive from your school district? 

c. Did you receive support from other sources? [if yes] In what ways and from whom? 

Note to interviewer: If necessary, probe the interviewee with the following questions to gather more 

information about his or her induction support. 

11. Please tell me more about your experiences with coaching or mentoring as a new 
principal. 

a. To what extent did you feel that the coaching or mentoring support you received as 
a new principal was beneficial to your practice? Would you say the support was not 

88 


at all beneficial, was somewhat beneficial, or was greatly beneficial? Please explain 
your answer. 

b. Probe (for principals with more than 2 pears of experience in their current role): Do you still 
receive any mentoring, coaching, peer network, professional development, or training 
support? [If yes] Could you please tell me about this support? 

Overall Perception of Preparedness 

12. Are there specific practices that you learned through [PROGRAM NAME] or supports 
provided by the district that have especially helped you be successful in your role as 
principal? 

a. Probe [If yes]: Coidd you please describe them and tell me how you have applied them in your 
principal practice? 

13. What do you feel was the most useful aspect of the training you received through 
[PROGRAM NAME]? Why was it valuable? 

14. What do you feel was the least useful aspect of the training you received through 
[PROGRAM NAME]? Please explain why it was the least useful. 

15. What do you feel was the most useful aspect of the support you received from your 
district as a new principal? Why was it valuable? 

16. What do you feel was the least useful aspect of the support you received from your 
district as a new principal? Please explain why it was the least useful. 

Conclusion 

17. Do you have any questions or comments you would like to add? 

Thank you for a rich discussion today. We really appreciate your time and the feedback you have 

given us. 
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Appendix E. Detailed Principal Perception Findings 

The results of the 68 interviews conducted with principals that graduated from the five selected 
programs included generally positive feedback on the programs. Principal respondents described 
program design and program supports that were both aligned to the selection criteria used to 
identify programs for participation in the study, and aligned with how programs described 
themselves through program documentation and interviews. The key findings from these 
interviews include the following: 

• All principals interviewed indicated that the selected principal preparation program that 
they attended was aligned to a specific set of research-based principal standards or 
competencies. When asked to what extent, the vast majority (90%) said their program was 
aligned to a set standards to "a great extent," and the remaining 10% of the respondents 
said to "a moderate extent." 

• Approximately 75% of the respondents found the coursework to be supportive of their 
practice in the principal role to "a great extent," with the majority also reporting that they 
applied their learning in the selected principal preparation program to their work in the 
principal role. 

• The respondents generally reported favorable perceptions of the residencies and 
internships they experienced during their preparation, noting the value of these hands-on 
experiences in preparing them for the principal role. In one program, participants were 
afforded two types of internship experience and may have had issues with one but overall 
favored the internship or residency experiences. 

• Ninety-nine percent of the respondents described multiple components and phases of the 
recruitment process (e.g., impromptu essays and scenario role-plays). Only one 
respondent described a straightforward interview process. In providing descriptions of 
the selection process, 9% of the respondents used the word rigorous, 6% of the respondents 
used the word intense, and 1% of the respondents described the selection process as 
extensive. 

• Across all selected principal preparation programs, the respondents described varied levels 
of support they received once they were in the principal role. Respondents from some 
selected programs noted that support for sitting principals was primarily offered through 
the district, with the respondents from one district describing a multiyear comprehensive 
support system. The respondents from other selected programs continued to receive 
coaching support from the program once they were in the principal role. They also 
described more informal supports that they received through colleagues and other leaders 
in the district. 

• Approximately half of the respondents (51% of the 41 respondents that provided an answer 
to this question) whose programs provided coaching support to new principals found the 
coaching support they received from their program to be "greatly beneficial" to their 
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practice in the principal role. Other respondents described the coaching provided as 
"somewhat beneficial" (37%), or only "beneficial" (7%). 

• When asked about the most useful aspects of the program, respondents had many favorable 
perceptions to share, noting that the most useful aspects of the selected principal 
preparation programs were are follows: 

Internship or residency 

Mentoring 

Coaching 

Focus on instructional leadership 
Reflections on the realities of the job of principal 
Cohort model and networking 
Role-playing and simulation exercises 

• The respondents reported variability in the district support they received once they 
assumed the role of principal. For example, one district had a structured program of 
support, whereas other districts offered optional support components or components that 
could be purchased after a provision period that was free of charge. 

• The respondents also had mixed perceptions of the usefulness of the district support with 
some respondents noting very minimal or ineffective district support, and others praising 
extensive support from the district. Some respondents also noted a lack of differentiation in 
district support. 

Perceived Alignment of Programs to Standards 

Across the five selected programs, the vast majority of the respondents said that the program in 
which they participated was aligned to a specific set of standards or competencies to a great 
extent, and these competencies were embedded throughout the program. Several respondents 
reported that the programs in which they participated consistently made them aware of the 
competencies they were expected to fulfill as part of their participation in the program and 
described the rubrics. According to the respondents, the competencies were related to 
instructional leadership, data-driven decision making, and creating a values-based culture. Of 
the few respondents who reported that their programs were aligned with a clear set of standards 
to a moderate extent, some noted they were in the initial cohort of enrollees at which time the 
program was not yet fully aligned with a clear set of standards. Another respondent reported that 
the program was moderately aligned to a clear set of standards because some standards were 
emphasized more than others. 

Perception on Extent to Which Coursework Supported Practice 

A majority of the interview respondents (74%) across all the programs said that the coursework 
of their program supported their practice as school leaders to a great extent. The respondents 
who indicated that the coursework supported their practice to a great extent said that the 
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coursework reflected the practical application of competencies and duties of an effective 
principal. One respondent explained, "I could do projects and papers studying problems in my 
school. What I would do was directly applicable to my school and ways to improve my school." 
Several interview respondents mentioned that the coursework activities provided opportunities 
to address challenging issues in a safe setting. The program participants applauded several 
aspects of their coursework, including group projects, case studies, role-playing activities, the 
book selection, resources, and the quality of course content. Several interview respondents also 
reported that they learned about a variety of instructional leadership frameworks and how to 
provide quality feedback to teachers. One respondent commented, "All those courses were and 
still are incredibly useful to my work." Interview respondents who said that coursework 
supported their practice as a principal to a moderate extent asserted that the coursework was 
more theoretical than practical. Some respondents who said that coursework supported their 
practice to a moderate extent said the residency experience was more valuable to them than the 
coursework. Other respondents reported that some courses were more relevant than others. 

Reflections on Experiential Learning Opportunities Within Programs 

Although most of the interview respondents across all programs reported participating in a 
residency or internship (referred to here as experiential learning opportunities), some 
participants from some programs reported not participating in any experiential learning 
component. Participants in two of the five programs noted that those in earlier cohorts were not 
exposed to experiential learning; feedback from those in earlier cohorts prompted a change, and 
an internship component was added to the program. Overall, the interview respondents who 
engaged in summer or yearlong residencies or internships reported that these experiential 
learning opportunities prepared them for the role of principal to a great extent. The respondents 
said that the experiential components were purposeful and aligned with standards. 

The respondents also expressed great appreciation for the experiential learning component, 
noting that it exposed them to a variety of facets of the principalship and afforded valid 
opportunities to learn from mistakes. One respondent noted the advantage of "being coached 
through a mistake before you are actually in the hot seat yourself." The participants described 
engagement in substantive responsibilities and tasks before actually becoming a principal. They 
reported being given autonomy related to specific projects, being given "real work," and "diving 
into big issues." The interview respondents for two of the programs described how the 
experiential learning component was designed to take them out of their comfort zone by 
assigning them a grade or school setting with which they were not familiar to make them well 
rounded. For example, one program offered two residencies, and there was a deliberate attempt 
to place aspiring principals in two very different schools for the two residencies. Often, the two 
residencies would differ in student enrollment, leadership style, and student demographics. 

Several respondents noted that one powerful component of the residency experience was being 
paired with a strong mentor principal. Several respondents across the five programs noted that 
the mentor principal often was strong in an area in which the aspiring principal was weak. One 
respondent who participated in internships or residencies and felt that the experiential learning 
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prepared him or her to be an effective principal to a moderate extent said that he or she felt that 
way because at the conclusion of the internship or residency, he or she was not prepared to be 
principal and needed more time as an administrator. Another respondent who said the 
experiential learning component prepared him or her to a moderate extent remarked, 

I don't think there's anything that can prepare you to a great extent. I don't think there's 
anything like just being in the job and doing the job, and having that responsibility . . . 
because I think there's still a very steep learning curve in that first year especially. 

Reflections on the Recruitment and Selection Process for Programs 

The majority of the respondents reported finding out about their programs through professional 
connections or word of mouth. Other applicants found out about their program through 
informational events, flyers, and online search or receiving an e-mail regarding the program. 

Across all five programs, the interview respondents consistently described the selection process for 
the program to which they were admitted using terms such as "extensive," "intense," "very 
rigorous," or "overall one of the best processes I've seen." The respondents reported that as 
program candidates they had to submit a written application, letters of recommendation, and a 
leadership portfolio. The application process included simulations, individual and group data 
analysis exercises, case study analysis, presentations, and impromptu essays. One respondent 
indicated that he or she could tell that he or she was being judged against the competencies that 
would comprise the program. Applicants reported participating in traditional interviews, panel 
interviews, and group interviews that required multiple applicants to work together. One 
respondent remarked, "The interview itself was multiple people representing different angles of 
what it might take to be a principal, including former principals, current principals, and education 
license staff." A component of the selection process for one program was described as follows: "We 
would be in a fish bowl, where there were observers all around us, and they would throw scenarios 
at us and let us brainstorm and have discussions as if we were a team working together." 

Perceptions of Early Career Supports 

Many of the interview respondents indicated that they received support once they completed the 
principal preparation program. Support in their new role as principal came from the program, 
the district, or other sources. 

Support From the Programs 

The interview respondents from two of the five programs indicated that their program did not 
offer structured or formal types of support, such as coaching or mentoring to graduates after they 
were placed as principals in the district. The respondents in these two programs mentioned they 
were still in touch with their professors and other cohort members, and some receive informal 
support from certain program graduates. The respondents who had participated in the other 
three programs mentioned that coaching support was provided through the program for a 
minimum of 1 year and up to 3 years. 
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Two participants in one program that provides coaching support said that although they felt 
supported by their program, they felt as if they received less support from their program than 
other program participants. The rationale they gave is that they have enrolled another school 
leadership program offered by the same organization that prepared them, which is designed to 
help develop school leadership teams. The program is based on competencies similar to those of 
their preparation program and comprises workshops as well as ongoing coaching. 

Participants in two of the five programs said that their principal preparation program supports 
principals through regular meetings of participants who have concluded the program's 
experiential component. For both programs, the meetings are designed to provide professional 
development and networking opportunities for program alumni. One respondent noted that 
attendance is not always high at these events, and the content program was not always relevant 
to participants' needs. Another respondent described how at such sessions participants would 
engage in professional conversations about what they were doing and how they were progressing 
in their role as principals. 

Support From the Districts 

All participants from one of the five programs described a 5-year comprehensive induction and 
development program offered by the district. Participants from this program offered positive 
remarks on the walk-throughs that were a part of the professional learning communities offered 
in Year 1 of this program and described another aspect of this 5-year development program that 
included individual leadership style development as "a really, really wonderful way of looking 
at ourselves as leaders." 

Support provided to participants in the remaining four programs was not as structured. The 
interview respondents from one program reported on support district support provided by 
network chiefs or executive directors. Views on this type of district support were mixed. 

In two of the five districts, the sentiment was that more early support is received from the 
program than from the district. One respondent remarked, "I would say that I've gotten more 
support from my [principal preparation program] coach than I have my network chief. They are 
just too busy; they are too overwhelmed trying to manage [a large number of] schools." 

Participants in one district mentioned that the superintendent provides monthly districtwide 
professional development. Other interview respondents mentioned that some districts provided 
coaches and mentors; however, it was not always consistent. For example, in one district, 
principals could opt to use funds from their school budget to obtain coaching from the district. 
The interview respondents in two districts mentioned receiving support from district networks 
or district leadership on budgeting, compliance issues, and human resources issues that many 
respondents found to be very beneficial. 
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Support From Other Sources 

Approximately 90% of the interview respondents reported receiving support or mentoring from 
an informal source that they sought out themselves. They said that strong bonds were developed 
with their cohort-mates while they were students and residents in their preparation programs. 
Many noted that veteran principals were happy to oblige requests for advice and assistance, and 
former professors and instructors also were supportive. In addition to informal support, the 
respondents mentioned receiving support from Teach For America through a leadership 
electronic mailing list, from a university executive leadership program, from the Gates 
Foundation, and from the Network for College Success. Some interview respondents mentioned 
receiving support from grant funding or support provided to principals of nontraditional types 
of schools, such as an international school, a charter school, or those opening a specially themed 
school. 

Perceptions of Coaching Supports 

The vast majority of the respondents from three programs described the coaching they received 
as greatly beneficial to their practice as a principal. (The respondents from the other two programs 
did not describe coaching support.) Program participants received between 1 and 5 years of 
coaching after becoming a principal. Some program participants had multiple coaches (e.g., a 
district provided coach or a coach provided by the program). A respondent from one program 
said. 

Every 2 weeks [my coach] comes to campus. We go into classrooms. I call it my therapy 
sessions where he comes in and he coaches me . . .. We may sit and talk about systems 
that need to be tweaked and get ideas. 

The respondents from another program described their interactions with program coaches who 
are retired administrators as effective. Another respondent from the same program described 
monthly visits by a coach from the program who provided strategic advice and helped the 
respondent parse through student data in ways in which he or she might not otherwise. Another 
respondent from the same program said that his or her coach came to the campus frequently 
during the first year as a principal, and the coach was on call 24/7 if he or she was struggling with 
an issue. 

The respondents from a different program who rated the coaching as greatly beneficial 
mentioned that the topics on which they received coaching support included organizing teams 
and learning communities, budgeting, hiring, classroom observations, data, and instructional 
strategies. The respondents mentioned that coaches held them accountable, were knowledgeable, 
asked pertinent questions, and helped them think through strategies and communication with 
staff. Coaches also helped in the review of ideas and documents; provided sound advice 
regarding teacher observations, school improvement, and other topics; and made frequent visits 
to the schools. Of one coach a respondent said, "I had tremendous support. [The coach] was top- 
notch, and I learned so much about what to do and what not to do." 
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A few principals in one program perceived the coaching support as somewhat beneficial to their 
practice as a principal. The rationale was that the coaching was on an as-needed basis. One 
respondent admitted being hesitant to reach out to a coach for assistance when situations 
occurred, preferring to take on the responsibility, knowing that help was available if needed. 
Another said that things went well the first year of the principalship, so the coaching was not 
intensive because "they didn't feel I needed that much support." One respondent, out of all 
program respondents, explained as follows: 

[Coaching has] not been beneficial at all because I don't feel like I'm receiving it. . . . It's 
just frustrating. . . I know 99% of the job. I feel like going into the second year, I don't 
know it well, but I know it. So, having support, or at least feedback of what I am doing 
well and what I'm not doing well, that would be appreciated. 

Perceptions of the Application of Program Learning to Principal Role 

The respondents provided numerous examples of how they applied what they learned through 
their principal preparation program. One respondent provided an example of applying what had 
been learned in the program by working with teachers in grade-level teams related to student 
work and standards-based outcomes to drive teachers to change their instructional practices. 

The respondents across three of the programs specifically identified practices that they applied 
that were aligned to the competencies of observation and providing effective feedback, data- 
driven instruction, and developing action plans. The interview respondents for two programs 
indicated that the preparation program provided them with a better and renewed understanding 
about the importance of being an instructional leader and coach in building school staff capacity. 
The respondents in one program noted that being part of that program helped them understand 
school leadership in an urban setting; the importance of critical thinking; how to impact 
organizational change; and how to develop a learning organization built on collaboration, skill 
sets, communication, and shared instructional practices. Four respondents in another program 
said that they applied systems analysis or organizational diagnosis as a practice that they learned 
in the program. 

The interview respondents also mentioned applying practices related to involving others in 
decision making, using protocols, supporting teachers, addressing organizational leadership, 
teamwork, and having difficult conversations with adults. Additional applied program learning 
mentioned by the respondents included developing a strong school climate or working 
conditions, being a reflective principal, and focusing on the instructional leadership role. One 
interview respondent remarked, "I guess I adopted a lot of practices I had not been exposed to 
before." 

Most Useful and Least Useful Aspects of the Programs 

The majority of the respondents in two of the five programs had a difficult time identifying one 
most useful aspect of the training program. These respondents mentioned more than one useful 
aspect of the training, with the rationale that all program components were "interrelated" and 
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"equally necessary and vital to my development." One respondent said, "[T]he coaching, 
residency, and coursework were mutually reinforcing in a way that any of the specific elements 
on their own wouldn't be. I do think that this is a case where the sum is greater than the value of 
its parts." Across the five programs, the respondents most frequently mentioned the following as 
the most useful aspects of their programs: 

• Internship or residency 

• Mentoring 

• Coaching 

• Focus on instructional leadership 

• Reflections on the realities of the job of principal 

• Cohort model and networking 

• Role-playing and simulation exercises 

Across the five programs, the respondents had a difficult time identifying the least useful aspects 
of their program. Many respondents opted not to provide an answer when asked about the least 
useful program aspects. One respondent remarked, "I honestly can't think of anything negative 
to say about the program. I can't rank anything as least because every project that I did was 
valuable." Of the respondents who did provide a response, those in three programs commented 
that one or two of the courses they took were not particularly useful. Course topics mentioned 
that were considered of little use included school finance, high-performing operational systems, 
research proposals, school law and policy, and psychology. 

Most Useful and Least Useful Aspects of Support From the Districts 

Interview data about the most useful aspects of support from the district were mixed across the 
five programs. Although the respondents praised one district for its extensive support, some 
respondents from another district declined to comment, saying that the support was so minimal 
that there was little to comment on regarding the most useful aspect. 

The respondents from four districts mentioned coaching support as valuable, whereas some 
participants from the fifth program mentioned that direct support from a district executive 
director or network staff was valuable. Some respondents from one program also expressed that 
district meetings afforded opportunities to interact with other principals, which is helpful. 
Several respondents mentioned that district professional development offerings on subjects such 
as budgeting and transferring funds, student information systems, or technology were useful. 
Additional district support mentioned as useful included operational and budgetary support, 
flexibility in the selection of a coach, district support specific to school needs, and support in 
helping principals understand district structure. The respondents in one district described the 
usefulness of their district convening of like-minded principals or principals that have similar 
schools into what one respondent to be "critical friends." 

Regarding least useful district-based supports, several respondents said the district made their 
job more difficult when they pulled them off campus for training or professional development. 
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Principals from two different programs said their district too frequently takes a "one-size-fits-all" 
approach to supporting principals and schools, and principals would benefit from more 
differentiated support especially because regions within the district have multiple schools and 
could not differentiate. In one program, several respondents mentioned that visits or support 
from local superintendents were the least useful aspect of support. 
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Appendix F. Detailed Quantitative Methods 

Model for Inexperienced Principals 

The model for the inexperienced principals analysis (RQ3) can be represented as follows: 

yigst = Po + X^-igstPl + X^-stPl + 

P 3 (year > 201 l) t * ( cohort = l) s + /? 4 * treatments * ( year > 201 l) t * ( cohort = l) s + 

p 5 (year > 2012 ) t * ( cohort = 2 ) s + /? 6 * treatments * ( year > 2012 ) t * ( cohort = 2 ) s + 

P 7 (year > 2013) t * ( cohort = 3) s + P 8 * treatments * (year > 2013) t * ( cohort = 3) s + 

P 9 (year > 2014) t * ( cohort = 4) s + /? 10 * treatments * (year > 2014) t * (cohort = 4) s + 

Y g + <P g * year t + u s +r s * year t + s igst (F-l) 

The models were estimated separately by district and subject (reading/ELA and mathematics) 
and assumed achievement y of student i in grade g in school s during year t as a function of 
individual student-level (XI i gs t) and time- varying (X2 s t) school-level covariates. These covariates 
differed across districts depending on availability. Table F-l shows the specific covariates for each 
district. The coefficient P 3 represents the average change in achievement in schools that received 
an inexperienced principal who was not trained by one of the selected programs in 2011-12, 
following the arrival of that new principal. The sum of the coefficients Ps and P 4 represents the 
change in average achievement among students in schools that received an inexperienced 
principal from one of the selected programs in 2011-12. The coefficients Pr, through P 10 represent 
the analogous effects for the other three cohorts (2012-13, 2013-14, and 2014-15) of inexperienced 
principals. The coefficients P 4 , P 6 , Pr, and P 10 , which represent average differences in effectiveness 
between treatment and comparison principals in the same cohort, are the parameters of interest. 


Table F-l. Covariates Included in the Statistical Models for Each District 


Model Covariates 

Models F-l to F-3 
(Inexperienced 
Principals) 

Models 

F-4 and 

F-5 

Models F-8 and 
F-9 (All 
Principals) 

District 

A 

B 

c 

D 

E 

A 

B 

c 

D 

Student is African American. 

X 

X 

X 

X 

X 





Student is Asian American. 

X 

X 

X 

X 

X 





Student is Hispanic. 

X 

X 

X 

X 

X 





Student is White. 

X 

X 

X 

X 

X 





Student belongs to another ethnic group. 

X 

X 

X 

X 

X 





(The most frequent of the five ethnicity 
categories is omitted.) 










Student's ethnicity is missing. 




X 

X 





Student is female. 

X 



X 






Gender is missing. 




X 






Student is an ELL. 

X 

X 

X 

X 

X 

X 

X 

X 

X 
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Model Covariates 

Models F-l to F-3 
(Inexperienced 
Principals) 

Models 

F-4 and 

F-5 

Models F-8 and 
F-9 (All 
Principals) 

District 

A 

B 

c 

D 

E 

A 

B 

c 

D 

Student is disabled (SWD). 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Student is eligible for free or reduced-price 
meals. 


X 

X 

X 

X 


X 

X 

X 

Indicator for free or reduced-price meals 
status is missing. 


X 



X 


X 



Grade indicators 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students who are African 
American 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students who are Asian 
American 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students who are Hispanic 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students who are White 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students with other ethnicity 

X 

X 

X 

X 

X 

X 

X 

X 

X 

(The most frequent of these five ethnicity 
categories is omitted.) 










Percentage of students with missing 
ethnicity 




X 

X 




X 

Percentage of students who are female 

X 





X 




Percentage of students who are ELLs 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students who are SWDs 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students who are eligible for 
free or reduced-price meals 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students with a missing free 
or reduced-price meals status 


X 



X 


X 



Percentage of students in Grade 4 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students in Grade 5 

X 

X 

X 

X 

X 

X 

X 

X 

X 

Percentage of students in Grade 6 



X 

X 

X 

X 

X 

X 

X 

Percentage of students in Grade 7 



X 

X 

X 

X 

X 

X 

X 

Percentage of students in Grade 8 



X 

X 

X 

X 


X 

X 

(Percentage of Students in Grade 3 is the 
omitted category.) 










Grade-specific linear time trends 

X 

X 

X 

X 

X 





Student changed schools between last year 
and this year, interacted with grade 
indicators. 






X 

X 

X 

X 
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Model Covariates 

Models F-l to F-3 
(Inexperienced 
Principals) 

Models 

F-4 and 

F-5 

Models F-8 and 
F-9 (All 
Principals) 

District 

A 

B 

c 

D 

E 

A 

B 

c 

D 

Student was retained in the same grade. 






X 

X 

X 

X 

Student has skipped a grade. 






X 

X 

X 

X 

Principal's first year of experience 






X 

X 

X 

X 

Principal's second year of experience 






X 

X 

X 

X 

Principal's third year of experience 






X 

X 

X 

X 

Principal's fourth year of experience 






X 

X 

X 

X 

Principal's fifth year of experience 






X 

X 

X 

X 

Principal's experience is missing 






X 


X 

X 

(Principal has more than 5 years of 
experience is the omitted category.) 










School fixed effects (excluded from Model 3) 

X 

X 

X 

X 

X 

X 

X 

X 

X 

School-specific linear trends 

X 

X 

X 

X 

X 





Year fixed effects 






X 

X 

X 

X 

Student fixed effects 






X 

X 

X 

X 


The models included grade-level fixed effects (y^) and grade-level trends (q> g * year t ) to account for 
the possibility that grade-specific levels and trends in student achievement among the schools 
included in the study differed from the overall levels and trends in student achievement within 
each grade in the district. In addition, the models included school fixed effects (u s ) to account for 
time-invariant differences in achievement between schools and school-specific (fixed) linear time 
trends (t s * year t ) to account for differential trends in achievement among schools. Standard errors 
(sigst) are clustered within each school-by-year cell. 

To obtain an overall estimate of the relative effectiveness of principals from selected programs 
across all cohorts, we estimated a simpler version of the model that estimates overall post and 
treatment times post effects across all cohorts of schools, which is presented in Equation F-2. 

yigst = Po + xi igstPr + X2 st p 2 + p 3 post st + /? 4 * treatments * post st + y g + (p g * year t + u s + 
r s * year t + e igst (F-2) 

In this model, the post indicator equals 1 in years following the arrival of an inexperienced principal 
at the school and 0 otherwise, and the binary treatment variable indicates that the principal who 
arrives in the post period was trained by one of the selected programs. The other model parameters 
are the same as in Equation F-l, and the standard errors are clustered at the school-by-year level. 
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Matching 

Table F-2 through Table F-5 present baseline test scores of schools receiving principals from 
selected programs (treatment schools) and comparison schools for Districts A, B, C, and D. To 
calculate the averages and SDs presented in these tables, we first calculated the average 
normalized student test score in each district, school, year, and subject. A simple average of these 
school-level averages was then presented for each district, cohort, and group (treatment or 
comparison). The SDs presented here are the SDs among schools, not among students. Because 
of the small number of potential comparison matches in Districts A, B, and C, the comparison 
schools included in the analyses for these districts do not match as closely to the treatment schools 
as the comparison schools included in the analysis for District D. 

In District A, the absolute value of the difference between the average test scores in treatment 
schools and average test scores in comparison schools ranged from 0.47 to 0.74 for the cohort of 
schools that received inexperienced principals in 2011-12, from 0.06 to 0.29 in the 2012-13 cohort, 
and from 0.01 to 0.12 in the 2013-14 cohort (see Table F-2). Treatment schools in the 2011-12 and 
2013-14 cohorts tended to have higher baseline test scores in District A, whereas in the 2012-13 
cohort, the comparison schools tended to have higher baseline test scores in District A. Because 
only one treatment school was in the 2012-13 cohort in District A, the SDs of average test scores 
for that school were not reported. 

In District B, the absolute value of the difference between the average test scores in treatment 
schools and average test scores in comparison schools ranged from 0.01 to 0.35 in mathematics 
and from 0.01 to 0.19 in reading (see Table F-3). In some years, cohorts, and subjects, the average 
baseline scores were higher in treatment schools, whereas in others, the average baseline scores 
were higher in comparison schools. Because only one treatment school was in the 2012-13 cohort 
in District B, the SDs of average test scores for that school were not reported. 

In District C, the absolute value of the difference between the average test scores in treatment 
schools and the average test scores in comparison schools ranged from 0.04 to 0.36 in mathematics 
and from 0.01 to 0.41 in reading (see Table F-4). In most years, cohorts, and subjects, the average 
baseline scores were lower in the treatment schools than in the comparison schools. 

In District D, the absolute values of the differences between the average test scores in treatment 
schools and the average test scores in comparison schools were smaller than in other districts, 
ranging from 0.01 to 0.14 (see Table F-5). The baseline achievement of schools that received 
inexperienced principals in 2011-12 was lower in schools that received principals from one of the 
selected programs. The average difference in achievement between the treatment and comparison 
schools also was larger for the 2011-12 cohort than for the other cohorts. The baseline 
achievement of schools that received inexperienced principals in 2012-13 or 2013-14 was higher 
in schools that received principals from one of the selected programs, although the differences 
were small, in all but one case less than 0.05 SD. 
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Table F-2. Baseline Test Score Comparison (School-Level): District A 


Year 

Cohort = 2011-12 

Cohort = 2012-13 

Cohort = 2013-14 

Average Normalized 
Mathematics Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

Comp. a 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

2008-09 

-0.184 

0.29 

-0.193 

0.382 

0.186 

-0.140 

0.187 

-0.076 

0.502 

0.092 

0.454 

0.055 

(.154) 

(.556) 

(.138) 

(.314) 

(.016) 

N/A 

(.011) 

N/A 

(.068) 

(.909) 

(.007) 

(.911) 

2009-10 

-0.278 

0.339 

-0.287 

0.454 

0.098 

-0.179 

0.13 

-0.088 

0.386 

0.171 

0.355 

0.121 

(0.160) 

(.559) 

(.062) 

(.343) 

(.211) 

N/A 

(.072) 

N/A 

(.039) 

(.687) 

(.005) 

(.871) 

2010-11 

-0.221 

0.362 

-0.141 

0.46 

0.057 

-0.098 

0.048 

-0.032 

0.373 

0.155 

0.410 

0.116 

(.073) 

(.468) 

(.321) 

(.279) 

(.144) 

N/A 

(.003) 

N/A 

(.065) 

(.589) 

(.006) 

(.760) 

2011-12 

Post 

0.059 

0.073 

0.048 

0.231 

0.416 

0.027 

0.356 

0.053 

(.312) 

N/A 

(.159) 

N/A 

(.040) 

(.640) 

(.128) 

(.727) 

2012-13 

Post 

0.342 

0.026 

0.276 

0.065 

(.201) 

(.734) 

(.152) 

(.709) 

2013-14 

Post 

2014-15 


Note. Scores were first averaged up to the school level within each year; a simple average across comparison and treatment schools was then calculated for each 
cohort and year. SD s are reported in parentheses. Comp. = comparison schools; Treat. = treatment schools; N/A indicates the SDs cannot be calculated (because the 
cell represents only one school). 


Table F-3. Baseline Test Score Comparison (School Level): District B 


Year 

Cohort = 2012-13 

Cohort = 2013-14 

Cohort = 2014-15 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

2008-09 

0.222 

0.523 

0.114 

0.043 

-0.388 

-0.206 

-0.370 

-0.260 

-0.231 

-0.113 

-0.295 

-0.105 

(.099) 

N/A 

(.113) 

N/A 

(.147) 

(.412) 

(.173) 

(.240) 

(.177) 

(.198) 

(.079) 

(.244) 

2009-10 

0.127 

0.372 

0.039 

0.049 

-0.215 

-0.192 

-0.369 

-0.225 

-0.143 

-0.176 

-0.244 

-0.145 

(.063) 

N/A 

(.086) 

N/A 

(.179) 

(.204) 

(.195) 

(.257) 

(.114) 

(.071) 

(.162) 

(.091) 

2010-11 

0.086 

0.278 

0.092 

0.014 

-0.212 

-0.255 

-0.396 

-0.456 

-0.017 

-0.155 

-0.218 

-0.357 

(.078) 

N/A 

(.18) 

N/A 

(.187) 

(.335) 

(.116) 

(.204) 

(.065) 

(.060) 

(.179) 

(.018) 

2011-12 

0.098 

0.444 

0.045 

-0.137 

-0.195 

0.100 

-0.359 

-0.268 

-0.167 

-0.192 

-0.224 

-0.264 

(.153) 

N/A 

(.149) 

N/A 

(.146) 

(.257) 

(.122) 

(.034) 

(.069) 

(.212) 

(.185) 

(.134) 

2012-13 

Post 

-0.259 

0.080 

-0.414 

-0.303 

-0.215 

-0.225 

-0.235 

-0.331 

(.180) 

(.103) 

(.173) 

(.097) 

(.133) 

(.032) 

(.162) 

(.062) 

2013-14 

Post 

-0.112 

-0.154 

-0.271 

-0.247 

(.071) 

(.310) 

(.129) 

(.158) 

2014-15 

Post 


Note. Scores were first averaged up to the school level within each year; a simple average across comparison and treatment schools was then calculated for each 
cohort and year. SDs are reported in parentheses. Comp. = comparison schools; Treat. = treatment schools; N/A indicates the SDs cannot be calculated (because the 
cell represents only one school). 
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Table F-4. Baseline Test Score Comparison (School Level): District C 


Year 

Cohort = 2011-12 

Cohort = 2012-13 

Cohort = 2013-14 

Cohort = 2014-15 

Average 

Normalized 

Mathematics 

Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics 

Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics 

Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics 

Score 

Average 
Normalized 
Reading Score 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

2008-09 

0.075 

-0.226 

-0.024 

-0.227 

0.280 

-0.049 

0.320 

-0.055 

0.079 

0.034 

0.050 

0.081 

-0.071 

-0.277 

-0.055 

-0.122 

(.354) 

(.200) 

(.370) 

(.122) 

(.343) 

(.441) 

(.370) 

(.524) 

(.545) 

(.252) 

(.491) 

(.236) 

(.297) 

(.196) 

(.262) 

(.103) 

2009-10 

-0.029 

-0.225 

-0.059 

-0.237 

0.228 

-0.067 

0.288 

-0.043 

0.075 

-0.016 

0.097 

0.039 

-0.081 

-0.23 

-0.018 

-0.108 

(.209) 

(.046) 

(.294) 

(.072) 

(.429) 

(.411) 

(.342) 

(.498) 

(.539) 

(.212) 

(.435) 

(.258) 

(.253) 

(.084) 

(.192) 

(.084) 

2010-11 

-0.009 

-0.099 

-0.033 

-0.152 

0.239 

-0.055 

0.319 

-0.092 

0.031 

-0.048 

0.050 

0.037 

-0.16 

-0.211 

-0.107 

-0.154 

(.263) 

(.136) 

(.324) 

(.120) 

(.411) 

(.344) 

(.362) 

(.386) 

(.556) 

(.250) 

(.460) 

(.306) 

(.365) 

(.134) 

(.342) 

(.066) 

2011-12 

Post 

0.252 

- 0.111 

0.296 

-0.043 

0.037 

-0.134 

0.040 

0.002 

-0.159 

-0.054 

-0.026 

-0.118 

(.454) 

(.313) 

(.334) 

(.336) 

(.512) 

(.222) 

(.431) 

(.295) 

(.323) 

(.188) 

(.278) 

(.167) 

2012-13 

Post 

0.035 

-0.075 

0.024 

-0.002 

-0.104 

-0.056 

-0.055 

-0.063 

(.533) 

(.187) 

(.443) 

(.220) 

(.273) 

(.289) 

(.296) 

(.035) 

2013-14 

Post 

-0.142 

-0.178 

-0.057 

-0.127 

(.206) 

(.137) 

(.222) 

(.090) 

2014-15 

Post 


Note. Scores were first averaged up to the school level within each year; a simple average across comparison and treatment schools was then calculated for each 
cohort and year. SDs are reported in parentheses. Comp. = comparison schools; Treat. = treatment schools. 
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Table F-5. Baseline Test Score Comparison (School Level): District D 


Year 

Cohort = 2011-12 

Cohort = 2012-13 

Cohort = 2013-14 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

Average 

Normalized 

Mathematics Score 

Average Normalized 
Reading Score 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

Comp. 

Treat. 

2008-09 

0.057 

-0.030 

0.027 

-0.084 

-0.018 

-0.001 

-0.026 

0.006 

0.015 

0.041 

0.008 

0.033 

(.527) 

(.369) 

(.402) 

(.338) 

(.380) 

(.403) 

(.364) 

(.342) 

(.650) 

(.336) 

(.529) 

(.407) 

2009-10 

0.020 

-0.073 

0.008 

-0.091 

-0.022 

-0.013 

-0.049 

-0.036 

0.068 

0.096 

0.041 

0.074 

(.544) 

(0.390) 

(.412) 

(.335) 

(.359) 

(.453) 

(.324) 

(.301) 

(.580) 

(.382) 

(.464) 

(.364) 

2010-11 

-0.036 

-0.180 

-0.035 

-0.173 

-0.026 

0.019 

-0.047 

-0.007 

0.121 

0.149 

0.106 

0.113 

(.555) 

(.532) 

(.416) 

(.384) 

(.417) 

-0.070 

(.417) 

-0.050 

(.405) 

-0.098 

(.344) 

-0.059 

(.567) 

0.047 

(.448) 

0.080 

(.508) 

0.113 

(.410) 

0.135 

2011-12 

Post 

(.429) 

(.431) 

(.437) 

(.361) 

(.620) 

(.412) 

(.573) 

(.412) 

2012-13 

Post 

0.052 

0.117 

0.091 

0.099 

(.602) 

(.491) 

(.621) 

(.486) 

2013-14 

Post 


Note. Scores were first averaged up to the school level within each year; a simple average across comparison and treatment schools was then calculated for each 
cohort and year. SDs are reported in parentheses. Comp. = comparison schools; Treat. = treatment schools. 
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Empirical Model— Variation in Effectiveness Among Principals 

In addition to examining average effects, to explore heterogeneity in effectiveness among 
inexperienced principals (RQ3a), we also estimated a separate post effect for each inexperienced 
principal who started at a school during the 2011-12, 2012-13, 2013-14, or 2014-15 school years, 
as follows: 

yigst = Po + XligstPi + X2 stp 2 +p 3 * ( cohort = l) s + . . . +/? 6 * ( cohort = 4) s + 

P 7 * ( cohort = l) s * ( ever treated) s + . . . +/? 10 * ( cohort = 4) s * ( ever treated) s + 

/in * ( cohort = l) s * year t + . . . +/? 14 * ( cohort = 4) s * year t + 

P 15 * ( cohort = l) s * ( ever treated) s * year t + . . . +/? 18 * ( cohort = 4) s * ( ever treated) s * 
year t + y g + (p g * year t + p s + 9 S * post st + S st + £ igst (F-3) 

These mixed models were estimated separately by district and subject and assumed achievement 
y of student i in grade g in school s during year t as a function of individual student-level (Xl, gst ) 
and school-level ( X2 st ) covariates, which differ across districts depending on availability (see 
Table F-l). The models included grade-level fixed effects (y g ) and grade-level trends (y> g * year t ) to 
account for the possibility that grade-specific levels and trends in student achievement among 
the schools included in the study differed from the overall levels and trends in student 
achievement within each grade in the district. 

The models included cohort fixed effects and fixed trends (for the four cohorts defined previously) 
to account for the possibility that achievement levels and trends in schools experiencing principal 
turnover in any year between 2011-12 and 2014—15 might be different, on average, from levels and 
achievement trends of schools in a district experiencing principal turnover in other years. In 
addition, the models interacted these cohort fixed effects and trends with time-invariant binary 
variables ( ever treated), indicating that the school received a principal from one of the selected 
programs. These interacted terms were included to allow for fixed differences in average levels and 
trends in student achievement between schools where principals from selected programs were 
placed and all other schools within each cohort. The models also allowed achievement at a school to 
shift up or down following the arrival of a new principal. 

Finally, these models estimated a random effect (p 5 ) for each school and a school-specific random 
coefficient (0?) on the post indictor— the binary variable that equals 1 during the years following 
the arrival of an inexperienced principal, which differs across cohorts, and 0 otherwise. The 
empirical Bayes estimates of these school-specific random coefficients on the post indicator 
represent our estimates of the contributions of individual inexperienced principals to student 
achievement. These estimates were assumed to be distributed with a mean of 0 and a variance of 
x; for this research question, we were primarily interested in the variance of x. These school- 
specific random coefficients were estimates of the regression-adjusted change in average student 
achievement following the arrival of the new, inexperienced principal. The models also included 
school-by-year (5 ?f ) random effects to account for the covariance of test scores among students 
attending the same school during the same year. 
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Empirical Model— District E 

The model describing the change in student achievement following the placement of each cohort 
of inexperienced principals in District E can be described as follows: 

yigst = Po + xi igstPi + X2 st p 2 + /? 3 (year > 2011) t * ( cohort = l) s + 

/3 4 (year > 2012) t * ( cohort = 2) s + /? 5 (year > 2013) t * ( cohort = 3) s + 

/?6 (year > 2014) t * ( cohort = 4) s + Yg + <Pg * year t + u s + t s * year t + e igst (F-4) 

The model was estimated separately by subject (reading/ELA and mathematics) and assumed 
achievement y of student i in grade g in school s during year t as a function of individual student- 
level {Xhgstj and time-varying (X2 st ) school-level covariates. Table F-l shows the specific 

covariates included for each district. The coefficient represents the average change in 

achievement in schools that received an inexperienced principal from one of the selected 
programs in 2011-12, following the arrival of that new principal; the coefficient P 4 represents the 
average change in achievement in schools that received an inexperienced principal from one of 
the selected programs in 2012-13, following the arrival of that new principal; and so on. The 
model included grade-level fixed effects (y, ? ) and grade-level trends (cp A , * year) to allow for the 
possibility that grade-specific levels and trends in student achievement among District E where 
principals from one of the selected programs were placed differed from the overall levels and 
trends in student achievement within each grade in the district. In addition, the models included 
school fixed effects ( u s ) to account for time-invariant differences in achievement between schools, 
and school-specific (fixed) linear time trends (is * year t ) to account for differential trends in 
achievement among schools. Standard errors are clustered within each school-by-year cell. 

To obtain an estimate of the average change in student achievement at a school across all cohorts 
following the placement of an inexperienced principal from one of the selected programs in 
District E, we estimated a simpler version of the model that estimated an overall post effect across 
all cohorts of schools (Equation F-5). 

yigst = Po+ XligstPi + X2 st p 2 + p 3 post st + Yg+(p g * y e ar t + u s + r s * year t + s igst (F-5) 

In this model, the post indicator equals 1 in years following the arrival of an inexperienced principal 
at the school and 0 otherwise. The other model parameters are the same as in Equation F-4, and the 
standard errors are clustered at the school-by-year level. 

Empirical Model— All Principals 

The exposure models, which were run separately by subject and district, can be represented as 
follows: 

yigst = Po + XligstPi + X2 st p 2 +p 3 * exposurel ist + p 4 * exposure 2 igst + * 

exposure3plus igst + u s + y g + + s ist (F-8) 

Achievement y of student i in grade g in school s during year t was modeled as a function of time- 
varying student characteristics (XI ,g St ), the characteristics of students attending the same grade 
and school as the student (X2 s t), and time-invariant average differences in student achievement 
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between schools (u s ), grades (y g ), and students (jn,). Standard errors are clustered by school and 
year. The models were implemented in Stata using the felsdvreg (fixed effects, least-squares, 
dummy variable regression) command (Cornelissen, 2008). 

The coefficients of interest are 03, P 4 , and (Is, which are the effect of having attended a school led 
by a treatment principal for one or more years, the additional effect of having attended a school 
led by a treatment principal for two or more years, and the additional effect of having attended a 
school led by a treatment principal for three or more years. We assumed that the effect of the 
treatment principal would be persistent. For example, if a student's achievement increased 
because of attending a school led by a treatment principal, then that increase in achievement 
would be permanent, even if the student did not attend a school led by a treatment principal in 
the subsequent year. Table F-6 is an example of how the exposure variables would be coded for a 
hypothetical student. 


Table F-6. Coding of Exposure Variables for a Student Attending a School Led by a Principal 
From One of the Selected Programs During the 2008-09 Through 2014-15 School Years 



School Year 

2008-09 

2009-10 

2010-11 

2011-12 

2013-14 

2014-15 

Current school led by treatment 
principal? 

No 

No 

Yes 

Yes 

Yes 

Yes 

exposurel 

0 

0 

1 

1 

1 

1 

exposure 2 

0 

0 

0 

1 

1 

1 

exposure3_plus 

0 

0 

0 

0 

1 

1 


The tenure models, which were run separately by subject and district, can be represented as 
follows: 

yigst = Po + XligstPi + X2 st p 2 + p 3 * tenure l st + /? 4 * tenure 2 st + p s * tenure_miss st + /? 6 * 
treatment st * tenure l st + /? 7 * treatment st * tenure 2 st + /? 8 * treatment st * tenure 3 st + u s + 
Y g +Pi + s ist (F-9) 

Achievement y of student i in grade g in school s during year t was modeled as a function of time- 
varying student characteristics (Xl, gst ), the characteristics of students attending the same grade and 
school as the student (X2 st ), and time-invariant average differences in student achievement between 
schools (its), grades (y g ), and students (p,). Standard errors are clustered by school and year. 

Student achievement was allowed to vary according to the number of years the principal had led 
the school. Specifically, we included binary variables for the principal's first year of tenure, the 
principal's second year tenure, and to indicate that principal tenure information was missing. 
Principals serving for three or more years at a school are the reference category. 

We also interacted principal tenure indicators with a binary variable indicating that a principal 
was from one of the selected programs. Estimates of 06 represent the contribution of treatment 
principals in their first year at a school to student achievement (relative to all other principals in 
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their first year of tenure), estimates of P 7 represent the contribution of treatment principals in their 
second year at a school to student achievement (relative to all other principals in their second 
year of tenure), and estimates of Ps represent the contribution of treatment principals with more 
than 2 years of tenure at a school to student achievement (relative to all other principals with 
more than 2 years of tenure). Because we were not missing tenure data for any principals from 
one of the selected programs, a term interacting tenure jniss st and treatment 5f was not included in 
the model. Table F-7 presents two examples of how the tenure variables would be coded for a 
hypothetical student. 


Table F-7. Coding of Tenure Variables for Two Hypothetical Students 



School Year 

2008-09 

2009-10 

2010-11 

2011-12 

2013-14 

2014-15 

Example 1 

Current school led by treatment 
principal? 

No 

No 

No 

Yes 

Yes 

Yes 

Year of principal tenure 

6th 

7th 

8th 

1st 

2nd 

3rd 

tenurel 

0 

0 

0 

1 

0 

0 

tenure2 

0 

0 

0 

0 

1 

0 

treatment * tenurel 

0 

0 

0 

1 

0 

0 

treatment * tenurel 

0 

0 

0 

0 

1 

0 

treatment * tenure3 

0 

0 

0 

0 

0 

1 

Example 2 

Current school led by treatment 
principal? 

No 

No 

No 

No 

No 

No 

Year of principal tenure 

1st 

2nd 

1st 

2nd 

3rd 

4th 

tenurel 

1 

0 

1 

0 

0 

0 

tenure2 

0 

1 

0 

1 

0 

0 

treatment * tenurel 

0 

0 

0 

0 

0 

0 

treatment * tenure2 

0 

0 

0 

0 

0 

0 

treatment * tenure3 

0 

0 

0 

0 

0 

0 
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Appendix G. Detailed Quantitative Findings 


Students and Schools Included in the Analysis of Inexperienced Principals 

District A Demographics 

Table G-l. Principal Experience in the Analysis of the Relative Impact of Inexperienced 
Principals From One of the Selected Programs for District A 
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Table G-2. Number of Students With Mathematics or Reading Test Scores and Average Normalized Mathematics and Reading 
Test Scores in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the Selected Programs for 
District A 




Number of Students 

Number of 

Mathematics Scores 

Number of Reading 
Scores 

Average Normalized 
Mathematics Score 

Average Normalized 
Reading Score 


i ear 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

3,506 

0 

3,504 

0 

3,490 

0 

0.151 

N/A 

0.147 

N/A 

2009-10 

3,432 

0 

3,429 

0 

3,418 

0 

0.165 

N/A 

0.153 

N/A 

2010-11 

3,538 

0 

3,538 

0 

3,519 

0 

0.165 

N/A 

0.168 

N/A 

2011-12 

2,732 

754 

2,732 

754 

2,725 

752 

0.095 

0.422 

0.077 

0.512 

2012-13 

2,294 

1,053 

2,294 

1,053 

2,287 

1,051 

0.044 

0.375 

0.055 

0.405 

2013-14 

1,744 

1,913 

1,743 

1,912 

1,740 

1,902 

0.073 

0.275 

0.069 

0.293 

2014-15 

1,551 

1,947 

1,551 

1,947 

1,551 

1,947 

0.060 

0.323 

0.047 

0.355 


Table G-3. Ethnicity of Students in Schools in Scores in Schools in the Analysis of the Relative Impact of Inexperienced 

Principals From One of the Selected Programs for District A 



Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 


Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 


2008-09 

5.5% 

N/A 

34.2% 

N/A 

12.3% 

N/A 

41.1% 

N/A 

6.9% 

N/A 


2009-10 

5.4% 

N/A 

33.9% 

N/A 

13.0% 

N/A 

41.6% 

N/A 

6.2% 

N/A 


2010-11 

5.9% 

N/A 

33.6% 

N/A 

13.1% 

N/A 

42.4% 

N/A 

5.1% 

N/A 


2011-12 

5.2% 

11.9% 

36.6% 

17.1% 

15.3% 

7.4% 

38.1% 

60.3% 

4.7% 

3.2% 


2012-13 

3.7% 

13.3% 

33.7% 

27.5% 

17.4% 

6.1% 

40.5% 

50.0% 

4.6% 

3.1% 


2013-14 

4.1% 

11.6% 

33.1% 

30.2% 

20.3% 

8.8% 

38.6% 

46.5% 

3.8% 

2.9% 


2014-15 

4.4% 

12.4% 

32.9% 

31.0% 

23.2% 

9.0% 

36.9% 

45.4% 

2.6% 

2.1% 
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Table G-4. Other Characteristics of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From 
One of the Selected Programs for District A 


Year 

Percentage Female 

Percentage ELL 

Percentage SWD 

Percentage Low Income 

Other 

Selected Program 

Other 

Selected Program 

Other 

Selected Program 

Other Selected Program 

2008-09 

49.0% 

N/A 

9.4% 

N/A 

12.1% 

N/A 

Not Available 

2009-10 

49.4% 

N/A 

11.0% 

N/A 

11.0% 

N/A 

2010-11 

48.5% 

N/A 

12.9% 

N/A 

10.1% 

N/A 

2011-12 

48.4% 

49.7% 

12.9% 

8.9% 

9.1% 

7.2% 

2012-13 

48.8% 

48.1% 

12.9% 

6.1% 

8.8% 

5.3% 

2013-14 

49.5% 

49.1% 

12.5% 

7.0% 

8.0% 

6.5% 

2014-15 

49.9% 

49.1% 

13.0% 

7.1% 

10.6% 

8.8% 


Table G-5. Enrolled Grade of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of 
the Selected Programs for District A 


Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

35.1% 

N/A 

32.9% 

N/A 

32.0% 

N/A 

Excluded 

Excluded 

Excluded 

2009-10 

34.7% 

N/A 

32.5% 

N/A 

32.8% 

N/A 

2010-11 

34.1% 

N/A 

33.2% 

N/A 

32.8% 

N/A 

2011-12 

34.0% 

33.8% 

33.6% 

34.6% 

32.5% 

31.6% 

2012-13 

31.9% 

32.1% 

34.5% 

34.8% 

33.6% 

33.1% 

2013-14 

32.9% 

35.8% 

32.9% 

32.0% 

34.2% 

32.2% 

2014-15 

34.9% 

33.4% 

34.4% 

34.9% 

30.8% 

31.7% 
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District B Demographics 

Table G-6. Number of Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the Selected 
Programs for District B 



Year 

Cohort 2012 

Cohort 2013 

Cohort 2014 

Cohort 2015 

Total Number of Schools 


Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2009-10 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2010-11 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2011-12 

0 

0 

6 

0 

9 

0 

7 

0 

22 

0 

2012-13 

0 

0 

5 

1 

9 

0 

7 

0 

21 

1 

2013-14 

0 

0 

5 

1 

7 

2 

7 

0 

19 

3 

2014-15 

0 

0 

5 

1 

7 

2 

5 

2 

17 

5 


Table G-7. Principal Experience in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the 
Selected Programs for District B 




Years of Experience as Principal 

Total Number 

of Schools 


Year 

1 

2 

3 

4 

5 

6 or More 

Missing 

2008-09 

3 

4 

7 

0 

2 

5 

1 

22 

2009-10 

2 

2 

5 

6 

0 

6 

1 

22 

2010-11 

3 

3 

1 

4 

5 

6 

0 

22 

2011-12 

4 

2 

3 

0 

5 

8 

0 

22 

2012-13 

7 

2 

1 

3 

0 

7 

2 

22 

2013-14 

9 

6 

0 

0 

1 

6 

0 

22 

2014-15 

7 

9 

6 

0 

0 

0 

0 

22 
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Table G-8. Number of Students With Mathematics or Reading Test Scores and Average Normalized Mathematics and Reading 
Test Scores in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the Selected Programs for 
District B 



Year 

Number of Students 

Number of 

Mathematics Scores 

Number of 
Reading Scores 

Average Normalized 
Mathematics Score 

Average Normalized 
Reading Score 


Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

4,825 

0 

4,805 

0 

3,801 

0 

-0.135 

N/A 

-0.196 

N/A 

2009-10 

4,876 

0 

4,861 

0 

3,218 

0 

-0.102 

N/A 

-0.219 

N/A 

2010-11 

4,855 

0 

4,840 

0 

2,723 

0 

-0.083 

N/A 

-0.237 

N/A 

2011-12 

4,878 

0 

4,810 

0 

2,732 

0 

-0.080 

N/A 

-0.223 

N/A 

2012-13 

4,554 

130 

4,514 

130 

2,702 

55 

-0.162 

-0.016 

-0.263 

-0.127 

2013-14 

4,210 

523 

4,171 

521 

2,398 

302 

-0.146 

-0.202 

-0.238 

-0.359 

2014-15 

3,710 

1,078 

3,691 

1,070 

2,280 

450 

-0.145 

-0.147 

-0.244 

-0.262 


Table G-9. Ethnicity of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the 
Selected Programs for District B 



Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 


Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

0.8% 

N/A 

27.8% 

N/A 

69.5% 

N/A 

0.6% 

N/A 

1.3% 

N/A 

2009-10 

0.5% 

N/A 

27.0% 

N/A 

70.4% 

N/A 

0.6% 

N/A 

1.5% 

N/A 

2010-11 

0.4% 

N/A 

25.8% 

N/A 

71.8% 

N/A 

0.6% 

N/A 

1.4% 

N/A 

2011-12 

0.9% 

N/A 

26.2% 

N/A 

71.4% 

N/A 

0.6% 

N/A 

0.9% 

N/A 

2012-13 

1.0% 

0.0% 

26.0% 

10.0% 

71.9% 

84.6% 

0.5% 

3.1% 

0.6% 

2.3% 

2013-14 

0.9% 

0.2% 

25.2% 

31.7% 

72.6% 

64.1% 

0.6% 

2.7% 

0.7% 

1.3% 

2014-15 

1.0% 

0.2% 

28.9% 

19.1% 

68.2% 

78.6% 

1.1% 

1.6% 

0.8% 

0.6% 
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Table G-10. Other Characteristics of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From 
One of the Selected Programs for District B 


Year 

Percentage Female 

Percentage ELL 

Percentage SWD 

Percentage Low 
Income 2 

Percentage Income 
Status Missing 

„ , Selected 

Other 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

Not Available 

43.9% 

N/A 

9.1% 

N/A 

94.6% 

N/A 

0.0% 

N/A 

2009-10 

47.1% 

N/A 

7.5% 

N/A 

96.0% 

N/A 

0.0% 

N/A 

2010-11 

54.2% 

N/A 

4.4% 

N/A 

96.9% 

N/A 

0.0% 

N/A 

2011-12 

54.6% 

N/A 

4.6% 

N/A 

97.2% 

N/A 

2.2% 

N/A 

2012-13 

55.3% 

64.6% 

3.9% 

8.5% 

97.4% 

98.5% 

1.7% 

1.5% 

2013-14 

55.5% 

47.6% 

3.7% 

5.9% 

96.6% 

96.0% 

2.1% 

1.3% 

2014-15 

51.3% 

63.5% 

3.9% 

5.2% 

95.6% 

95.4% 

3.3% 

3.4% 


“Eligible for free or reduced-price meals. Students with missing information were coded as being not from low-income families. 


Table G-ll. Enrolled Grade of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One 
of the Selected Programs for District B 


Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

37.5% 

N/A 

31.5% 

N/A 

31.0% 

N/A 

Excluded 

Excluded 

Excluded 

2009-10 

35.5% 

N/A 

33.7% 

N/A 

30.8% 

N/A 

2010-11 

33.7% 

N/A 

34.2% 

N/A 

32.2% 

N/A 

2011-12 

33.9% 

N/A 

32.2% 

N/A 

33.9% 

N/A 

2012-13 

34.7% 

39.2% 

33.5% 

30.0% 

31.8% 

30.8% 

2013-14 

35.0% 

27.2% 

33.1% 

36.1% 

31.9% 

36.7% 

2014-15 

34.7% 

35.7% 

33.6% 

32.6% 

31.8% 

31.7% 
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District C Demographics 

Table G-12. Number of Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the Selected 
Programs for District C 



Year 

Cohort 2012 

Cohort 2013 

Cohort 2014 

Cohort 2015 

Schools 


Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

8 

0 

11 

0 

13 

0 

9 

0 

41 

0 

2009-10 

8 

0 

11 

0 

13 

0 

9 

0 

41 

0 

2010-11 

8 

0 

11 

0 

13 

0 

9 

0 

41 

0 

2011-12 

5 

3 

11 

0 

13 

0 

9 

0 

38 

3 

2012-13 

5 

3 

7 

4 

13 

0 

9 

0 

34 

7 

2013-14 

5 

3 

7 

4 

9 

4 

9 

0 

30 

11 

2014-15 

5 

3 

7 

4 

9 

4 

7 

2 

28 

13 


Table G-13. Principal Experience in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the 
Selected Programs for District C 



Year 

Years of Experience as Principal 

Total Number of 

Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2008-09 

3 

4 

5 

2 

8 

19 

0 

41 

2009-10 

3 

3 

4 

4 

2 

24 

1 

41 

2010-11 

2 

4 

3 

3 

3 

24 

2 

41 

2011-12 

12 

2 

3 

2 

1 

20 

1 

41 

2012-13 

12 

12 

2 

3 

1 

11 

0 

41 

2013-14 

13 

11 

10 

1 

1 

5 

0 

41 

2014-15 

9 

13 

11 

8 

0 

0 

0 

41 
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Table G-14. Number of Students With Mathematics or Reading Test Scores and Average Normalized Mathematics and Reading 
Test Scores in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the Selected Programs for 
District C 



Year 

Number of Students 

Number of 

Mathematics Scores 

Number of 
Reading Scores 

Average Normalized 
Mathematics Score 

Average Normalized 
Reading Score 


Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

9,072 

N/A 

9,040 

N/A 

8,828 

N/A 

0.085 

N/A 

0.087 

N/A 

2009-10 

9,444 

N/A 

9,418 

N/A 

9,242 

N/A 

0.055 

N/A 

0.081 

N/A 

2010-11 

9,572 

N/A 

9,533 

N/A 

9,389 

N/A 

0.046 

N/A 

0.076 

N/A 

2011-12 

9,051 

640 

9,023 

637 

8,854 

635 

0.046 

-0.134 

0.091 

-0.094 

2012-13 

8,193 

1,412 

8,173 

1,409 

8,046 

1,337 

0.045 

-0.076 

0.081 

-0.024 

2013-14 

7,486 

2,230 

7,461 

2,227 

7,310 

2,096 

0.042 

-0.051 

0.088 

-0.009 

2014-15 

6,907 

2,676 

6,829 

2,656 

6,672 

2,497 

0.020 

-0.035 

0.039 

-0.018 


Table G-15. Ethnicity of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the 
Selected Programs for District C 



Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 


Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

3.9% 

N/A 

14.3% 

N/A 

54.6% 

N/A 

20.6% 

N/A 

6.5% 

N/A 

2009-10 

3.6% 

N/A 

13.2% 

N/A 

52.4% 

N/A 

21.4% 

N/A 

9.5% 

N/A 

2010-11 

3.6% 

N/A 

13.0% 

N/A 

52.6% 

N/A 

21.0% 

N/A 

9.8% 

N/A 

2011-12 

3.4% 

3.4% 

13.5% 

2.3% 

51.9% 

79.5% 

23.0% 

8.3% 

8.2% 

6.4% 

2012-13 

3.2% 

2.8% 

14.4% 

4.5% 

52.1% 

71.0% 

23.3% 

14.9% 

6.9% 

6.9% 

2013-14 

3.1% 

2.5% 

15.0% 

5.7% 

51.8% 

67.7% 

23.9% 

18.0% 

6.3% 

6.1% 

2014-15 

2.8% 

2.8% 

15.1% 

9.2% 

51.4% 

66.3% 

25.5% 

16.5% 

5.1% 

5.2% 
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Table G-16. Other Characteristics of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From 
One of the Selected Programs for District C 


Year 

Percentage Female 

Percentage ELL 

Percentage SWD 

Percentage Low Income 3 

Other Selected Program 

Other 

Selected Program 

Other 

Selected Program 

Other 

Selected Program 

2008-09 

Not Available 

28.5% 

N/A 

14.8% 

N/A 

70.8% 

N/A 

2009-10 

32.4% 

N/A 

12.4% 

N/A 

65.5% 

N/A 

2010-11 

32.5% 

N/A 

12.3% 

N/A 

71.3% 

N/A 

2011-12 

33.5% 

38.9% 

12.3% 

14.8% 

70.8% 

90.6% 

2012-13 

34.2% 

44.8% 

11.9% 

13.2% 

71.3% 

83.4% 

2013-14 

32.9% 

38.3% 

11.6% 

12.0% 

72.7% 

79.1% 

2014-15 

29.0% 

34.4% 

11.6% 

12.2% 

72.1% 

78.1% 


“Eligible for free or reduced-price meals. Students with missing information were coded as being not from low-income families. 


Table G-17. Enrolled Grade of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One 
of the Selected Programs for District C 


Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

24.8% 

N/A 

21.7% 

N/A 

22.0% 

N/A 

11.4% 

N/A 

10.2% 

N/A 

9.9% 

N/A 

2009-10 

24.5% 

N/A 

23.4% 

N/A 

20.5% 

N/A 

12.4% 

N/A 

9.8% 

N/A 

9.5% 

N/A 

2010-11 

23.8% 

N/A 

23.4% 

N/A 

22.3% 

N/A 

10.4% 

N/A 

10.7% 

N/A 

9.5% 

N/A 

2011-12 

24.0% 

17.3% 

23.6% 

17.0% 

22.4% 

13.4% 

10.2% 

17.5% 

9.5% 

15.8% 

10.3% 

18.9% 

2012-13 

23.4% 

25.3% 

23.3% 

23.9% 

22.5% 

24.7% 

10.6% 

10.8% 

10.0% 

8.5% 

10.1% 

6.8% 

2013-14 

23.9% 

29.1% 

22.5% 

25.0% 

21.4% 

26.2% 

10.4% 

8.9% 

10.6% 

5.7% 

11.2% 

5.1% 

2014-15 

23.0% 

28.1% 

23.0% 

28.6% 

21.8% 

26.2% 

11.3% 

6.9% 

10.1% 

5.8% 

10.8% 

4.3% 
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District D Demographics 

Table G-18. Number of Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the Selected 
Programs for District D 



Table G-19. Principal Experience in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the 
Selected Programs for District D 
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Table G-20. Number of Students With Mathematics or ELA Test Scores and Average Normalized Mathematics and ELA Test 
Scores in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One of the Selected Programs for 
District D 


Year 

Number of Students 

Number of 

Mathematics Scores 

Number of ELA Scores 

Average Normalized 
Mathematics Score 

Average Normalized 
ELA Score 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

27,245 

N/A 

27,054 

N/A 

26,450 

N/A 

0.049 

N/A 

0.011 

N/A 

2009-10 

28,448 

N/A 

28,345 

N/A 

27,584 

N/A 

0.031 

N/A 

0.000 

N/A 

2010-11 

28,633 

N/A 

28,479 

N/A 

27,853 

N/A 

0.034 

N/A 

0.003 

N/A 

2011-12 

26,012 

2,239 

25,895 

2,227 

25,513 

2,171 

0.021 

-0.209 

0.002 

-0.239 

2012-13 

23,881 

4,399 

23,828 

4,394 

23,349 

4,288 

-0.004 

-0.097 

0.004 

-0.102 

2013-14 

21,853 

6,202 

21,148 

6,161 

21,442 

6,044 

-0.012 

0.011 

-0.018 

-0.013 

2014-15 

21,447 

6,156 

20,966 

6,138 

20,943 

5,990 

-0.007 

-0.010 

-0.020 

-0.050 


Table G-21. Ethnicity of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From Selected 
Programs for District D 


Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage 

Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 

Percentage 
Ethnicity Missing 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

16.1% 

N/A 

27.5% 

N/A 

39.4% 

N/A 

14.2% 

N/A 

0.3% 

N/A 

2.5% 

N/A 

2009-10 

16.3% 

N/A 

26.6% 

N/A 

38.7% 

N/A 

14.3% 

N/A 

0.3% 

N/A 

3.8% 

N/A 

2010-11 

15.9% 

N/A 

25.5% 

N/A 

39.1% 

N/A 

14.3% 

N/A 

0.3% 

N/A 

4.8% 

N/A 

2011-12 

15.6% 

19.5% 

23.0% 

38.4% 

40.2% 

36.7% 

15.4% 

1.6% 

0.4% 

0.3% 

5.4% 

3.6% 

2012-13 

15.7% 

20.3% 

22.2% 

29.1% 

40.2% 

40.2% 

16.5% 

5.6% 

0.6% 

0.3% 

4.8% 

4.4% 

2013-14 

15.7% 

24.3% 

21.8% 

23.9% 

41.1% 

37.9% 

16.7% 

9.9% 

1.0% 

0.7% 

3.8% 

3.4% 

2014-15 

16.1% 

26.3% 

21.2% 

21.8% 

42.0% 

38.8% 

16.8% 

9.9% 

1.3% 

1.1% 

2.6% 

2.1% 
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Table G-22. Other Characteristics of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From 
One of the Selected Programs for District D 


Year 

Percentage Female 

Percentage Gender 
Missing 

Percentage ELL 

Percentage SWD 

Percentage 

Low Income 3 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

2008-09 

48.9% 

N/A 

0.1% 

N/A 

15.3% 

N/A 

9.4% 

N/A 

63.5% 

N/A 

2009-10 

48.6% 

N/A 

0.2% 

N/A 

14.9% 

N/A 

17.8% 

N/A 

85.3% 

N/A 

2010-11 

48.9% 

N/A 

0.2% 

N/A 

15.8% 

N/A 

17.8% 

N/A 

86.0% 

N/A 

2011-12 

49.0% 

48.0% 

0.2% 

0.1% 

14.2% 

22.8% 

18.8% 

17.0% 

83.9% 

98.5% 

2012-13 

49.2% 

47.9% 

0.2% 

0.1% 

13.7% 

17.8% 

17.7% 

16.8% 

77.1% 

84.1% 

2013-14 

49.3% 

48.2% 

0.2% 

0.0% 

11.2% 

13.6% 

21.0% 

19.1% 

76.4% 

77.7% 


“Eligible for free or reduced-price meals. Students with missing information were coded as being not from low-income families. 


Table G-23. Enrolled Grade of Students in Schools in the Analysis of the Relative Impact of Inexperienced Principals From One 
of the Selected Programs for District D 


Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 

Other 

Selected 

Program 

Other 

Selected 

Program 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

Selected 

Program 

Other 

2008-09 

17.7% 

N/A 

18.0% 

N/A 

18.0% 

N/A 

16.6% 

N/A 

15.6% 

N/A 

14.0% 

N/A 

2009-10 

17.3% 

N/A 

18.1% 

N/A 

17.5% 

N/A 

15.6% 

N/A 

16.0% 

N/A 

15.5% 

N/A 

2010-11 

17.0% 

N/A 

18.2% 

N/A 

17.7% 

N/A 

15.5% 

N/A 

15.5% 

N/A 

16.1% 

N/A 

2011-12 

16.2% 

30.6% 

16.5% 

31.4% 

17.0% 

30.0% 

16.7% 

2.7% 

16.4% 

2.8% 

17.1% 

2.5% 

2012-13 

15.9% 

29.5% 

16.5% 

27.3% 

15.3% 

27.0% 

17.5% 

4.7% 

17.6% 

5.5% 

17.3% 

6.0% 

2013-14 

15.3% 

25.0% 

16.0% 

28.9% 

15.2% 

26.3% 

17.4% 

5.9% 

18.0% 

6.5% 

18.1% 

7.4% 

2014-15 

15.5% 

23.4% 

15.7% 

29.2% 

15.4% 

28.6% 

17.1% 

6.3% 

17.9% 

6.0% 

18.4% 

6.5% 
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Results— Changes in Student Achievement in District E 

Demographics in District E 

Only schools that received a principal from one of the selected programs were included in our 
analysis of District E. More than 5,000 students in Grades 3-8 with reading or mathematics scores 
were enrolled in these schools each year between 2008-09 and 2013-14, but the total number of 
students fell to about 4,800 in 2014-15 (Table G-24). Student test scores in these schools were very 
close to the district average from 2008-09 to 2012-13. In 2013-14, the average student test scores 
in these schools was about 0.02 SD below the district average in reading and about 0.06 SD below 
the district average in mathematics. In 2014-15, the average student test scores in these schools 
was about 0.20 SD below the district average in reading and about 0.23 SD below the district 
average in mathematics (Table G-24). 


Table G-24. Number of Students With Mathematics or Reading Test Scores and Average 
Normalized Mathematics and Reading Test Scores in Schools in the District E Analysis 


Year 

Number of 

Students 

Number of 

Mathematics 

Scores 

Number of 
Reading 
Scores 

Average 

Normalized 

Mathematics Score 

Average 
Normalized 
Reading Score 

2008-09 

5,752 

5,731 

5,709 

0.014 

0.011 

2009-10 

5,659 

5,653 

5,619 

- 0.032 

0.002 

2010-11 

5,470 

5,462 

5,454 

- 0.022 

0.001 

2011-12 

5,437 

5,431 

5,414 

0.008 

0.007 

2012-13 

5,320 

5,307 

5,303 

0.004 

0.016 

2013-14 

5,092 

5,081 

5,042 

- 0.019 

- 0.064 

2014-15 

4,814 

4,772 

4,721 

- 0.195 

- 0.227 


On average between 2008-09 and 2014-15, among students in District E schools where a principal 
from one of the selected programs was placed in 2011-12, 2012-13, 2013-14, or 2014-15, 45% of the 
students were Hispanic, 37% were African American, 11% were White, and 2% were Asian American 
(Table G-25). On average between 2008-09 and 2013-14, 12% of the students in these schools were 
ELLs, 13% were SWDs, and 76% were from low-income families (Table G-25). In 2014-15, almost 15% 
of the students were ELLs, 15% were SWDs, and nearly 84% were from low-income families. The 
increase in the share of students who were ELLs, SWDs, or from low-income families might explain 
some of the decrease in average test scores in these schools in 2014—15. On average between 2008-09 
and 2014—15, among students in District E schools where a principal from one of the selected 
programs was placed in 2011-12, 2012-13, 2013-14, or 2014—15, 56% of the students were in 
Grades 3-5, 16% were in Grade 6, and 28% were in Grades 7 and 8 (Table G-26). 


123 



Table G-25. Characteristics of Students in Schools in the District E Analysis 



Year 

Percentage 

Asian 

American 

Percentage 

African 

American 

Percentage 

Hispanic 

Percentage 

White 

Percentage 
Other Ethnic 
Group 

Percentage 

Ethnicity 

Missing 

Percentage 

ELL 

Percentage 

SWD 

Percentage 

Low 

Income 


2008-09 

1.9% 

38.1% 

44.9% 

10.9% 

0.2% 

4.0% 

11.3% 

13.0% 

75.4% 

2009-10 

1.8% 

37.1% 

45.3% 

10.5% 

0.2% 

5.1% 

10.8% 

13.1% 

78.3% 

2010-11 

1.9% 

36.3% 

45.4% 

10.7% 

0.5% 

5.2% 

12.7% 

13.1% 

74.8% 

2011-12 

2.0% 

36.2% 

45.4% 

11.2% 

0.9% 

4.3% 

13.0% 

13.5% 

76.0% 

2012-13 

2.3% 

36.2% 

45.8% 

11.4% 

1.0% 

3.3% 

12.9% 

13.7% 

75.1% 

2013-14 

2.4% 

37.8% 

44.6% 

11.5% 

1.1% 

2.6% 

12.5% 

14.2% 

75.0% 

2014-15 

2.6% 

37.8% 

46.6% 

10.7% 

0.0% 

2.3% 

14.6% 

15.1% 

83.5% 


Table G-26. Enrolled Grade of Students in Schools in the District E Analysis 




Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 


2008-09 

18.7% 

17.5% 

17.6% 

16.7% 

14.2% 

15.3% 

2009-10 

20.1% 

17.8% 

17.6% 

16.4% 

14.0% 

14.0% 

2010-11 

19.3% 

18.4% 

17.8% 

16.2% 

14.1% 

14.1% 

2011-12 

20.0% 

18.3% 

17.9% 

15.9% 

14.2% 

13.8% 

2012-13 

18.9% 

18.7% 

18.0% 

15.8% 

14.5% 

14.2% 

2013-14 

19.5% 

18.6% 

18.5% 

15.8% 

14.1% 

13.5% 

2014-15 

20.3% 

18.4% 

18.0% 

17.7% 

13.5% 

12.0% 
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All Principals Model Results 

Students and Schools Included in the Analysis of All Principals 

Table G-27. Years of Experience Among Principals Who Were Not Trained by One of the Selected Programs in the Analysis of All 
Principals for District A 



Year 

Years of Experience as Principal — Comparison Principals 

Total Number of 

Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

20 

14 

16 

18 

15 

43 

15 

141 

2010-11 

14 

22 

11 

13 

15 

51 

13 

139 

2011-12 

9 

13 

17 

12 

13 

52 

8 

124 

2012-13 

14 

8 

14 

14 

13 

52 

8 

123 

2013-14 

18 

14 

8 

10 

12 

56 

4 

122 

2014-15 

23 

16 

12 

9 

9 

52 

2 

123 


Table G-28. Years of Experience Among Principals Who Were Trained by One of the Selected Programs in the Analysis of All 
Principals for District A 



Year 

Years of Experience as Principal— Treatment Princi 

rals 

Total Number of 

Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

0 

0 

0 

0 

0 

0 

0 

0 

2010-11 

2 

0 

0 

0 

0 

0 

0 

2 

2011-12 

5 

3 

0 

0 

0 

0 

0 

8 

2012-13 

1 

5 

3 

0 

0 

0 

0 

9 

2013-14 

2 

1 

5 

3 

0 

0 

0 

11 

2014-15 

1 

2 

1 

5 

2 

0 

0 

11 
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Table G-29. Students' Cumulative Number of Years Attending a School Led by a Principal Trained by One of the Selected 
Programs in the Analysis of All Principals for District A 


Year 

Number of Years of Exposure 

Total Number of Students 

0 

1 

2 

3 

4 

2009-10 

56,094 

0 

0 

0 

0 

56,094 

2010-11 

56,900 

422 

0 

0 

0 

57,322 

2011-12 

54,302 

2,320 

221 

0 

0 

56,843 

2012-13 

51,335 

2,061 

1,329 

93 

0 

54,818 

2013-14 

54,275 

3,307 

1,448 

629 

1 

59,659 

2014-15 

53,544 

4,814 

1,982 

894 

79 

61,234 


Table G-30. Number of Students With Mathematics or Reading Test Scores and Average Normalized Mathematics and Reading 
Test Scores Among Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis of 
All Principals for District A 


Year 

Number of Mathematics 

Scores 

Number of Reading Scores 

Average Normalized 
Mathematics Score 

Average Normalized 
Reading Score 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

2009-10 

56,004 

0 

55,676 

0 

0.046 

N/A 

0.039 

N/A 

2010-11 

56,813 

420 

56,417 

417 

0.052 

-0.370 

0.040 

-0.391 

2011-12 

54,209 

2,539 

53,937 

2,524 

0.040 

-0.146 

0.035 

-0.128 

2012-13 

51,225 

3,481 

51,008 

3,462 

0.036 

-0.104 

0.032 

-0.088 

2013-14 

54,197 

5,379 

54,011 

5,348 

0.051 

-0.075 

0.042 

-0.059 

2014-15 

53,512 

7,766 

53,506 

7,766 

0.034 

0.055 

0.025 

0.071 
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Table G-31. Ethnicity of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis 
of All Principals for District A 



Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 


Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

4.4% 

N/A 

39.8% 

N/A 

15.8% 

N/A 

34.5% 

N/A 

5.5% 

N/A 

2010-11 

4.6% 

5.0% 

39.8% 

47.4% 

16.4% 

29.6% 

33.9% 

11.1% 

5.2% 

6.9% 

2011-12 

4.9% 

6.0% 

39.4% 

38.4% 

17.3% 

24.8% 

33.4% 

25.9% 

5.0% 

4.8% 

2012-13 

4.8% 

6.9% 

38.8% 

39.6% 

18.3% 

23.5% 

33.3% 

25.4% 

4.9% 

4.6% 

2013-14 

5.2% 

7.5% 

39.2% 

38.9% 

19.1% 

23.0% 

32.5% 

27.1% 

4.0% 

3.6% 

2014-15 

5.3% 

7.7% 

39.6% 

34.6% 

21.3% 

20.1% 

30.6% 

34.7% 

3.2% 

3.0% 


Table G-32. Other Characteristics of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in 
the Analysis of All Principals for District A 



Year 

Percentage Female 

Percentage ELL 

Percentage SWD 

Percentage Low Income 


Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

50.1% 

N/A 

13.7% 

N/A 

10.3% 

N/A 

Not Available 

2010-11 

50.0% 

46.2% 

14.5% 

28.4% 

9.5% 

9.7% 

2011-12 

50.2% 

49.2% 

13.9% 

22.2% 

9.0% 

7.8% 

2012-13 

50.2% 

48.3% 

12.6% 

17.5% 

8.6% 

7.1% 

2013-14 

50.1% 

49.0% 

12.2% 

15.2% 

8.7% 

7.1% 

2014-15 

49.3% 

48.8% 

12.7% 

12.1% 

10.1% 

8.8% 
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Table G-33. Enrolled Grade of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the 
Analysis of All Principals for District A 


Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

18.4% 

N/A 

17.7% 

N/A 

16.9% 

N/A 

16.0% 

N/A 

15.6% 

N/A 

15.4% 

N/A 

2010-11 

18.0% 

32.5% 

17.5% 

34.1% 

17.0% 

33.4% 

16.0% 

0.0% 

15.9% 

0.0% 

15.5% 

0.0% 

2011-12 

17.3% 

31.4% 

16.8% 

32.6% 

16.4% 

31.4% 

16.9% 

4.5% 

16.3% 

0.0% 

16.3% 

0.0% 

2012-13 

15.8% 

22.2% 

17.1% 

27.4% 

16.6% 

27.0% 

16.3% 

20.2% 

17.3% 

3.2% 

16.9% 

0.0% 

2013-14 

17.1% 

24.1% 

15.8% 

21.9% 

16.4% 

24.0% 

16.4% 

15.9% 

16.9% 

12.2% 

17.4% 

1.8% 

2014-15 

18.1% 

16.1% 

16.6% 

22.1% 

15.2% 

20.7% 

16.1% 

17.1% 

16.5% 

13.1% 

17.4% 

10.9% 


Table G-34. Years of Experience Among Principals Who Were Trained by One of the Selected Programs in the Analysis of All 
Principals for District B 
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Table G-35. Years of Experience Among Principals Who Were Not Trained by One of the Selected Programs in the Analysis of All 

Principals for District B 


Year 

Years of Experience as Principal — Comparison Principals 

Total Number of Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

25 

18 

27 

35 

1 

87 

8 

201 

2010-11 

34 

18 

15 

24 

27 

74 

13 

205 

2011-12 

40 

21 

17 

13 

23 

79 

14 

207 

2012-13 

37 

24 

18 

16 

8 

86 

18 

207 

2013-14 

55 

17 

19 

14 

10 

75 

15 

205 

2014-15 

47 

38 

15 

16 

11 

62 

20 

209 


Table G-36. Students' Cumulative Number of Years Attending a School Led by a Principal Trained by One of the Selected 

Programs in the Analysis of All Principals for District B 


Year 

Number of Years of Exposure 

Total Number of Students 


0 

1 

2 

3 

2009-10 

53,973 

0 

0 

0 

53,973 

2010-11 

54,638 

0 

0 

0 

54,638 

2011-12 

54,967 

0 

0 

0 

54,967 

2012-13 

54,851 

257 

0 

0 

55,108 

2013-14 

54,656 

629 

127 

0 

55,412 

2014-15 

54,363 

1,148 

323 

17 

55,851 
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Table G-37. Number of Students With Mathematics or Reading Test Scores and Average Normalized Mathematics and Reading 
Test Scores Among Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis of 
All Principals for District B 


Year 

Number of Mathematics 

Scores 

Number of Reading Scores 

Average Normalized 
Mathematics Score 

Average Normalized Reading 
Score 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

2009-10 

53,708 

0 

43,643 

0 

0.001 

N/A 

0.000 

N/A 

2010-11 

54,446 

0 

41,040 

0 

0.000 

N/A 

0.000 

N/A 

2011-12 

54,156 

0 

41,373 

0 

0.001 

N/A 

-0.001 

N/A 

2012-13 

54,119 

257 

41,583 

175 

-0.001 

0.173 

-0.001 

0.113 

2013-14 

53,876 

754 

40,746 

505 

0.002 

-0.092 

0.002 

-0.110 

2014-15 

53,629 

1,479 

40,490 

831 

0.003 

-0.102 

0.037 

-0.109 


Table G-38. Ethnicity of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis 
of All Principals for District B 


Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

0.7% 

N/A 

24.9% 

N/A 

68.5% 

N/A 

4.1% 

N/A 

1.9% 

N/A 

2010-11 

0.6% 

N/A 

23.7% 

N/A 

70.0% 

N/A 

3.8% 

N/A 

1.8% 

N/A 

2011-12 

1.1% 

N/A 

23.1% 

N/A 

70.3% 

N/A 

3.9% 

N/A 

1.7% 

N/A 

2012-13 

1.0% 

0.0% 

22.4% 

7.8% 

71.1% 

77.0% 

4.0% 

13.2% 

1.4% 

1.9% 

2013-14 

1.1% 

0.4% 

22.4% 

24.5% 

71.2% 

66.8% 

4.2% 

7.0% 

1.2% 

1.3% 

2014-15 

1.2% 

0.2% 

22.4% 

19.6% 

71.0% 

75.8% 

4.4% 

3.6% 

1.0% 

0.7% 
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Table G-39. Other Characteristics of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in 
the Analysis of All Principals for District B 


Year 

Percentage Female 

Percentage ELL 

Percentage SWD 

Percentage Low 
Income 

Percentage Income 
Status Missing 

NOt T? . 

„ , Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

Not Available 

36.6% 

N/A 

8.6% 

N/A 

90.9% 

N/A 

0.0% 

N/A 

2010-11 

42.0% 

N/A 

4.9% 

N/A 

91.6% 

N/A 

0.0% 

N/A 

2011-12 

45.7% 

N/A 

4.3% 

N/A 

92.0% 

N/A 

7.1% 

N/A 

2012-13 

47.4% 

45.1% 

3.7% 

7.4% 

91.9% 

83.3% 

6.9% 

15.6% 

2013-14 

48.3% 

43.9% 

3.4% 

6.0% 

91.0% 

90.7% 

6.1% 

5.6% 

2014-15 

49.8% 

57.3% 

4.0% 

5.0% 

87.7% 

93.2% 

9.9% 

4.8% 


Table G-40. Enrolled Grade of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the 
Analysis of All Principals for District B 


Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

23.3% 

N/A 

21.4% 

N/A 

19.7% 

N/A 

18.1% 

N/A 

17.5% 

N/A 

N/A 

N/A 

2010-11 

22.4% 

N/A 

22.0% 

N/A 

20.6% 

N/A 

17.5% 

N/A 

17.5% 

N/A 

N/A 

N/A 

2011-12 

21.7% 

N/A 

21.4% 

N/A 

21.4% 

N/A 

18.3% 

N/A 

17.2% 

N/A 

N/A 

N/A 

2012-13 

21.6% 

38.5% 

20.9% 

33.5% 

20.1% 

28.0% 

19.5% 

0.0% 

17.9% 

0.0% 

N/A 

N/A 

2013-14 

22.1% 

25.5% 

21.0% 

33.9% 

20.0% 

33.3% 

18.1% 

7.3% 

18.8% 

0.0% 

N/A 

N/A 

2014-15 

22.3% 

26.1% 

21.3% 

28.1% 

20.1% 

28.9% 

18.2% 

13.6% 

18.1% 

3.4% 

N/A 

N/A 
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Table G-41. Years of Experience Among Principals Who Were Not Trained by One of the Selected Programs in the Analysis of All 
Principals for District C 



Year 

Years of Experience as Principal — Comparison Principals 

Total Number of Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

6 

9 

13 

9 

5 

47 

6 

95 

2010-11 

9 

9 

9 

9 

7 

44 

6 

93 

2011-12 

15 

8 

5 

8 

6 

41 

14 

97 

2012-13 

14 

16 

5 

6 

6 

31 

13 

91 

2013-14 

13 

14 

13 

3 

5 

30 

12 

90 

2014-15 

14 

15 

11 

10 

3 

27 

15 

95 


Table G-42. Years of Experience Among Principals Who Were Trained by One of the Selected Programs in the Analysis of All 
Principals for District C 



Year 

Years of Experience as Principal— Treatment Principals 

Total Number of Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

5 

2 

2 

2 

0 

0 

0 

11 

2010-11 

3 

5 

2 

2 

2 

0 

0 

14 

2011-12 

3 

3 

3 

2 

2 

2 

0 

15 

2012-13 

5 

4 

5 

3 

2 

4 

0 

23 

2013-14 

4 

5 

4 

5 

2 

5 

0 

25 

2014-15 

4 

5 

5 

4 

5 

3 

0 

26 
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Table G-43. Students' Cumulative Number of Years Attending a School Led by a Principal Trained by One of the Selected 
Programs in the Analysis of All Principals for District C 


Year 

Number of Years of Exposure 

Total Number of 

Students 

0 

1 

2 

3 

4 

5 

6 

2009-10 

26,365 

1,981 

657 

0 

0 

0 

0 

29,003 

2010-11 

25,672 

2,207 

1,150 

314 

0 

0 

0 

29,343 

2011-12 

25,299 

2,452 

1,356 

702 

47 

0 

0 

29,809 

2012-13 

23,266 

3,822 

1,660 

1,001 

111 

26 

0 

29,749 

2013-14 

22,751 

3,759 

2,350 

1,281 

150 

54 

1 

30,141 

2014-15 

21,999 

3,913 

2,317 

1,760 

114 

48 

0 

29,989 


Table G-44. Number of Students With Mathematics or Reading Test Scores and Average Normalized Mathematics and Reading 
Test Scores Among Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis of 
All Principals for District C 


Year 

Number of Mathematics 

Scores 

Number of Reading Scores 

Average Normalized 
Mathematics Score 

Average Normalized 
Reading Score 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

2009-10 

25,423 

3,474 

24,896 

3,270 

-0.005 

-0.009 

-0.023 

0.001 

2010-11 

24,603 

4,622 

24,181 

4,403 

-0.010 

-0.027 

-0.022 

-0.009 

2011-12 

23,824 

5,911 

23,384 

5,676 

0.003 

-0.017 

-0.020 

-0.048 

2012-13 

22,485 

7,283 

22,080 

6,991 

0.025 

-0.069 

0.001 

-0.104 

2013-14 

22,313 

7,922 

21,713 

7,617 

0.034 

-0.064 

0.010 

-0.091 

2014-15 

21,624 

8,006 

21,223 

7,795 

0.025 

-0.169 

0.025 

-0.154 
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Table G-45. Ethnicity of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis 
of All Principals for District C 



Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 


Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

3.5% 

2.0% 

13.9% 

11.2% 

54.3% 

61.8% 

19.5% 

17.1% 

8.7% 

7.9% 

2010-11 

3.7% 

2.0% 

13.6% 

10.5% 

54.3% 

63.0% 

19.1% 

16.9% 

9.4% 

7.6% 

2011-12 

3.7% 

2.2% 

13.7% 

9.4% 

54.2% 

65.1% 

20.3% 

17.0% 

8.1% 

6.5% 

2012-13 

3.8% 

2.0% 

13.8% 

9.5% 

52.8% 

66.9% 

22.6% 

16.0% 

7.1% 

5.6% 

2013-14 

3.8% 

2.1% 

13.5% 

9.6% 

52.3% 

65.9% 

24.0% 

17.1% 

6.4% 

5.3% 

2014-15 

3.7% 

2.1% 

13.7% 

10.3% 

51.3% 

68.5% 

25.8% 

14.7% 

5.5% 

4.4% 


Table G-46. Other Characteristics of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in 
the Analysis of All Principals for District C 



Year 

Percentage Female 

Percentage ELL 

Percentage SWD 

Percentage Low 
Income 


Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 


31.4% 

38.9% 

12.4% 

12.0% 

63.5% 

69.5% 

2010-11 


32.1% 

38.4% 

11.9% 

12.2% 

71.5% 

74.6% 

2011-12 

Not Available 

33.6% 

40.1% 

12.4% 

12.5% 

72.1% 

76.3% 

2012-13 

33.5% 

42.2% 

12.0% 

12.7% 

71.4% 

79.3% 

2013-14 


33.6% 

40.2% 

11.4% 

12.5% 

68.7% 

79.0% 

2014-15 


30.3% 

37.1% 

11.8% 

13.1% 

68.8% 

81.1% 
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Table G-47. Enrolled Grade of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the 
Analysis of All Principals for District C 



Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 


Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

18.7% 

30.2% 

17.7% 

25.6% 

17.1% 

19.7% 

15.6% 

11.4% 

15.3% 

6.9% 

15.7% 

6.2% 

2010-11 

18.3% 

28.6% 

18.2% 

25.1% 

17.4% 

20.5% 

14.5% 

12.3% 

15.9% 

8.5% 

15.6% 

5.0% 

2011-12 

18.7% 

26.7% 

18.3% 

23.9% 

18.0% 

19.4% 

14.2% 

12.0% 

14.7% 

10.5% 

16.1% 

7.6% 

2012-13 

19.3% 

23.6% 

18.6% 

23.5% 

17.7% 

21.3% 

14.2% 

12.5% 

14.7% 

9.9% 

15.5% 

9.2% 

2013-14 

21.0% 

21.0% 

19.2% 

22.8% 

17.7% 

22.3% 

13.0% 

13.9% 

14.3% 

11.5% 

14.8% 

8.5% 

2014-15 

21.2% 

18.8% 

20.5% 

21.4% 

18.3% 

22.4% 

13.2% 

14.9% 

12.8% 

12.0% 

14.0% 

10.5% 


Table G-48. Years of Experience Among Principals Who Were Not Trained by One of the Selected Programs in 
Principals for District D 

the Analysis of All 



Year 

Years of Ex 

rerience as Principal — Comparison Principals 

Total Number of Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

60 

68 

83 

95 

100 

538 

112 

1,056 

2010-11 

73 

63 

73 

82 

84 

564 

132 

1,071 

2011-12 

96 

69 

59 

67 

72 

582 

138 

1,083 

2012-13 

101 

100 

70 

59 

54 

583 

160 

1,127 

2013-14 

110 

98 

96 

68 

50 

572 

173 

1,167 

2014-15 

Data Not Available 
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Table G-49. Years of Experience Among Principals Who Were Trained by One of the Selected Programs in the Analysis of All 
Principals for District D 



Year 

Years of Experience as Principal — Treatment Principals 

Total Number of Schools 


1 

2 

3 

4 

5 

6 or More 

Missing 

2009-10 

27 

31 

21 

38 

35 

28 

0 

180 

2010-11 

22 

29 

32 

21 

31 

56 

0 

191 

2011-12 

13 

19 

33 

29 

18 

80 

0 

192 

2012-13 

17 

14 

19 

29 

23 

86 

0 

188 

2013-14 

14 

16 

13 

16 

25 

99 

0 

183 

2014-15 

Data Not Available 


Table G-50. Students' Cumulative Number of Years Attending a School Led by a Principal Trained by One of the Selected 
Programs in the Analysis of All Principals for District D 



Year 

Number of Years of Exposure 

Total Number of 

Students 


0 

1 

2 

3 

4 

5 

6 

2009-10 

373,105 

35,431 

31,929 

0 

0 

0 

0 

440,465 

2010-11 

365,571 

37,343 

25,144 

15,888 

0 

0 

0 

443,946 

2011-12 

362,481 

37,729 

25,447 

17,599 

4,513 

0 

0 

447,769 

2012-13 

363,329 

34,524 

26,635 

20,275 

3,489 

2,110 

0 

450,362 

2013-14 

368,672 

32,453 

23,888 

23,239 

3,554 

1,364 

868 

454,038 

2014-15 

Data Not Available 
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Table G-51. Number of Students With Mathematics or ELA Test Scores and Average Normalized Mathematics and ELA Test 
Scores Among Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis of All 
Principals for District D 


Year 

Number of Mathematics 

Scores 

Number of ELA Scores 

Average Normalized 
Mathematics Score 

Average Normalized ELA 
Score 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

Not Exposed 

Exposed 

2009-10 

349,439 

89,144 

340,100 

87,082 

0.034 

-0.142 

0.051 

-0.163 

2010-11 

346,748 

94,573 

339,777 

93,266 

0.048 

-0.126 

0.063 

-0.140 

2011-12 

350,654 

95,111 

344,895 

94,139 

0.040 

-0.138 

0.057 

-0.153 

2012-13 

358,137 

91,166 

352,087 

90,183 

0.036 

-0.140 

0.041 

-0.158 

2013-14 

358,392 

82,502 

361,024 

84,159 

0.030 

-0.127 

0.033 

-0.141 

2014-15 

Data Not Available 


Table G-52. Ethnicity of Students Exposed or Not Exposed to Principals Trained by One of the Selected Programs in the Analysis 
of All Principals for District D 


Year 

Percentage Asian 
American 

Percentage African 
American 

Percentage Hispanic 

Percentage White 

Percentage Other 
Ethnic Group 

Percentage 
Ethnicity Missing 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

14.9% 

11.2% 

28.4% 

36.2% 

38.4% 

40.4% 

14.7% 

8.4% 

0.4% 

0.3% 

3.1% 

3.5% 

2010-11 

15.1% 

10.9% 

27.6% 

35.5% 

38.6% 

40.3% 

14.7% 

8.9% 

0.3% 

0.4% 

3.7% 

4.0% 

2011-12 

15.2% 

10.9% 

27.0% 

34.8% 

38.7% 

40.0% 

15.1% 

9.7% 

0.4% 

0.4% 

3.7% 

4.2% 

2012-13 

15.4% 

11.6% 

26.6% 

33.8% 

39.0% 

39.8% 

15.2% 

10.4% 

0.6% 

0.5% 

3.3% 

4.0% 

2013-14 

15.5% 

12.4% 

26.4% 

32.4% 

39.4% 

40.0% 

15.2% 

11.0% 

0.8% 

0.7% 

2.6% 

3.6% 

2014-15 

Data Not Available 
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Table G-53. Other Characteristics of Students Exposed or Not Exposed to Principals Trained by One of the Selected Program in 
the Analysis of All Principals for District D 


Year 

Percentage Female 

Percentage Gender 
Missing 

Percentage ELL 

Percentage SWD 

Percentage Low Income 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

48.8% 

48.2% 

0.2% 

0.2% 

13.9% 

14.5% 

17.3% 

19.4% 

84.3% 

89.2% 

2010-11 

48.9% 

48.4% 

0.2% 

0.2% 

14.7% 

14.7% 

17.2% 

19.8% 

85.1% 

89.6% 

2011-12 

48.8% 

48.3% 

0.2% 

0.2% 

13.9% 

13.6% 

17.7% 

20.2% 

83.5% 

86.9% 

2012-13 

48.8% 

48.2% 

0.2% 

0.2% 

13.2% 

12.6% 

16.9% 

19.0% 

76.6% 

78.7% 

2013-14 

48.8% 

48.2% 

0.2% 

0.2% 

10.8% 

10.5% 

19.8% 

22.1% 

75.2% 

76.1% 

2014-15 

Data Not Available 


Table G-54. Enrolled Grade of Students Exposed or Not Exposed to Principals Trained by One of the Selected Program in the 
Analysis of All Principals for District D 


Year 

Grade 3 

Grade 4 

Grade 5 

Grade 6 

Grade 7 

Grade 8 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

Not 

Exposed 

Exposed 

2009-10 

16.9% 

16.5% 

16.5% 

18.4% 

16.1% 

17.9% 

15.7% 

18.9% 

16.9% 

14.6% 

17.8% 

13.7% 

2010-11 

18.0% 

14.1% 

16.9% 

16.2% 

16.6% 

17.9% 

15.6% 

19.1% 

15.9% 

18.1% 

17.1% 

14.6% 

2011-12 

19.0% 

11.4% 

17.3% 

14.4% 

16.6% 

16.7% 

15.8% 

20.1% 

15.4% 

19.2% 

15.9% 

18.2% 

2012-13 

19.0% 

11.4% 

18.0% 

12.0% 

16.8% 

15.6% 

15.6% 

19.7% 

15.4% 

20.9% 

15.1% 

20.3% 

2013-14 

19.2% 

10.5% 

17.9% 

12.5% 

17.4% 

13.3% 

15.4% 

20.1% 

15.1% 

21.1% 

15.1% 

22.5% 

2014-15 

Data Not Available 
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Effects of All Principals on Student Achievement by District 

District A 

Reading and Mathematics. In District A, we found no evidence that students exposed to 
principals from one of the selected programs scored any higher or lower in reading or mathematics 
than they would have scored if they had not been exposed to a principal from one of the selected 
programs (see Table 15, Figure 15, and Figure 16). Similarly, we found no evidence that reading or 
mathematics achievement improved when students attended a school led by a principal from one 
of the selected programs, relative to reading and mathematics achievement among students who 
attended schools led by principals with similar years of tenure (see Table 16 and Figure G-l.). 


Figure G-l. Impact of District A Principals on Student Achievement, by Tenure at School 



Note. SPPP = selected principal preparation program. 

District B 


Reading. In District B, where we observed only six principals from one of the selected programs, 
students exposed to principals from one of the selected programs for two or more years 
experienced a statistically significant 0.11 SD increase in reading achievement (see Table 15 and 
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Figure 15). However, the reading achievement of District B students who attended a school led 
by a principal from one of the selected programs did not increase when compared with the 
reading achievement of District B schools led by principals with similar tenures (see Table 16 and 
Figure G-2). This seemingly incongruous result can be explained as follows. Our exposure models 
assumed that the impact of principals on student achievement was permanent, whereas our 
tenure models did not. If the relative effectiveness of principals from one of selected programs 
was not apparent until one or more years after the student had been exposed to a principal from 
one of the selected programs, one would expect to observe significant differences in the exposure 
model but not the tenure model. 

Mathematics. In District B, we found no evidence that students exposed to principals from one of 
the selected programs scored any higher in mathematics than they would have scored if they had 
not been exposed to a principal from one of the selected programs (see Table 15 and Figure 16). 
Similarly, we saw no evidence that District B students who attended a school led by a principal 
from one of the selected programs experienced a change in mathematics achievement that differed 
from changes in mathematics achievement among other students in District B who attended a 
school led by a principal with the same years of tenure (see Table 16 and Figure G-2). 


Figure G-2. Impact of District B Principals on Student Achievement, by Tenure at School 



Note. SPPP = selected principal preparation program. 
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District C 

Reading. In District C, we found no evidence that students exposed to principals from one of the 
selected programs scored any higher in reading than they would have scored if they had not been 
exposed to a principal from one of the selected programs (see Table 15 and Figure 15). Similarly, 
we found no evidence that reading achievement improved when students who attended a school 
led by a principal from one of the selected programs, relative to reading achievement among 
students who attended schools led by principals with similar years of tenure (see Table 16 and 
Figure G-3). 

Mathematics. In District C, we found no evidence that students exposed to principals from one of 
the selected programs scored any higher in mathematics than they would have scored if they had 
not been exposed to a principal from one of the selected programs (see Table 15 and Figure 16). 
Similarly, we saw no evidence that District C students who attended a school led by a principal 
from one of the selected programs experienced a change in mathematics achievement that differed 
from changes in mathematics achievement among other students in District C who attended a 
school led by a principal with the same years of tenure (see Table 16 and Figure G-3). 


Figure G-3. Impact of District C Principals on Student Achievement, by Tenure at School 



Note. SPPP = selected principal preparation program. 
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District D 

ELA. In District D, we found evidence that students who were exposed to principals from one of 
the selected programs scored about 0.02 SD lower in ELA than they would have if they had not 
been exposed to a principal from one of the selected programs (see Table 15 and Figure G-4). 
However, although we found that students attending schools led by principals from selected 
programs who have two or more years of tenure experienced lower gains in ELA achievement 
than students in schools led by principals from other programs (see Table 16 and Figure G-4), the 
differences were not statistically significant. Taken together, these results suggested that 
District D principals from one of the selected programs may have been less effective at fostering 
ELA achievement than District D principals from other programs. 

Mathematics. In District D, we found evidence that students who were exposed to principals 
from one of the selected programs scored about 0.03 SD lower in mathematics than they would 
have if they had not been exposed to a principal from one of the selected programs (see Table 15 
and Figure 16). We found evidence that students fell even further behind in mathematics (about 
0.025 SD) if they were exposed to a principal from one of the selected programs for three or more 
years. We found no evidence that students attending schools led by principals from one of the 
selected programs in their first, second, third, or greater than third year of tenure experienced 
larger or smaller gains in mathematics achievement than students attending schools led by 
principals from other programs with similar years of tenure (see Table 16 and Figure G-4). These 
mixed mathematics results for District D may be consistent if students attending schools led by 
principals from one of the selected programs experienced short-term (or no) gains in achievement 
in mathematics relative to their peers, yet ultimately fell behind their peers as a result of having 
attended a school led by a principal from one of the selected programs. 
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Figure G-4. Impact of District D Principals on Student Achievement, by Tenure at School 



Note. SPPP = selected principal preparation program. 
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